代码上下文为王：Augment 面向专业软件工程师的 AI 助手，与 Guy Gur-Ari 对谈

Today my guest is Guy Gur-Ari, co-founder and chief scientist at Augment, a company using the full range of AI strategies—from autocomplete to RAG to chatbots to autonomous coding agents—to transform the practice of software engineering in large enterprise codebases.

While our first episodes in the Software Supernova series looked at vibe-coding platforms that allow anyone to prompt their way from zero to a proof of concept or basic app, Augment, which was founded in 2022, back when OpenAI’s Codex models and early autocomplete tools were still mostly just foreshadowing a very different way to code, is tackling a harder but potentially more economically transformative challenge: How do you 10x productivity for professional engineers who bring their considerable human expertise to bear on vast, messy, legacy codebases, which often have millions of lines of code spread across multiple projects that can vary in age, coding style, and underlying technical infrastructure?

Unlike personal projects, where one can often simply copy an entire codebase into Gemini’s context window, as Guy explains, the enterprise challenge requires serious technical firepower at all levels of the stack. Augment has spent the last 3 years deeply exploring multiple approaches to code understanding and has ultimately developed a sophisticated, retrieval-heavy approach from the ground up. Their RAG stack includes a custom-built vector database capable of real-time updates, proprietary retrieval models designed specifically for large codebases, code search that fires on every single keystroke for every single user, and custom code-generation models trained with a technique they call reinforcement learning from developer behavior. It also includes multiple product paradigms for delivering code to users, all of which are intensively optimized for both accuracy and speed and available across a number of the most popular development environments.

The results are quite impressive. As you’ll hear, Guy reports that he personally hasn’t written a line of code in months. These days, the coding agent, which I had the chance to use in preview and which will be released to the public very soon, handles all of that, leaving Guy to focus on higher-level issues, including how he and the team can continue to improve the agent so that it can eventually run for extended periods, take on larger projects, and even go beyond explicit user instructions to infer and address unstated needs.

The economics of the business are fascinating, too. Augment’s pricing is pretty conventional today, with $30 and $60 per month plans, but Guy was quite candid about the fact that some power users already cost them a whole lot more than that to serve. Especially as workflows consume more and more compute, pricing in the AI space in general is very much a live question. It helps, of course, to design pricing that aligns company and customer interests, but it’s less clear how best to do that, considering that enterprise customers also value stable pricing and predictable costs. The good news for Augment is that, having raised some $250 million in investment capital, they do have some time and financial cushion to figure that out.

There is a ton of technical depth in this episode, but arguably the most valuable part is Guy’s practical, down-to-earth advice for AI builders. While he and the Augment team have repeatedly invented new technology to solve hard problems, he recommends starting new projects simply by creating small evaluation datasets of just 10 to 20 high-quality, hand-labeled examples that you understand deeply and can quickly test new solutions against. He also recommends optimizing for the speed of iteration by pursuing the simplest available strategies first and then exhausting what’s available in the market before building custom solutions in-house. As regular listeners will know, I wholeheartedly endorse all of that advice.

Toward the end, I asked Guy if Augment is currently hiring junior engineers and, more broadly, what advice he has for today’s early-career engineers and computer science students. His answer, I think you’ll agree, reflects the current moment in the software industry: a sense of excitement and opportunity for the foreseeable future, but also a recognition that nobody can see the future more than 2 to 3 years out.

I hope you enjoy this deep dive into the hard tech powering AI coding assistance for enterprise software engineers with Guy Gur-Ari, co-founder and chief scientist at Augment.

Guy Gur-Ari, welcome to The Cognitive Revolution.

Great to be here. Thanks for having me.

My pleasure. I’m excited about this. We’ve been doing a little series that I’m calling the Software Supernova, which is just a nod to how much the software industry is changing. We’re coming at that from a bunch of different angles to try to understand it as deeply as possible.

I think you’re going to provide a really differentiated and interesting angle, because a lot of the stuff that we’ve looked at previously has been people who want to create an app out of nothing. There’s a growing number of products out there now that can take you quite far if you show up with just an idea and want to go from prompt to app in seconds, which is often the promise.

But you guys are coming at the software industry from basically the other end, targeting larger organizations with big codebases and things in production—long-lived projects. I think this will be a really interesting compare-and-contrast to understand the different challenges that poses and the different solutions you’re bringing to the market.

Maybe, for starters, give me a little introduction to the company and that core challenge. Folks who follow this feed are paying attention to AI, but I’m not actually sure how many have been in larger software organizations and would be familiar with the particular challenges that those organizations face.

For sure. Augment was founded with that vision of bringing AI to bear on real software engineering challenges that show up when you’re working on a large team and when you’re working on a large, existing codebase, because those are the challenges that the vast majority of professional developers face day to day.

Definitely, the larger the organization and the larger the codebase, the fewer zero-to-one projects people do and the more ongoing maintenance they handle. There’s still feature development and product development, but it all has to work in the context of a large codebase.

Our premise was that we could see AI technology crossing a threshold where it was becoming useful. Back when Augment was founded, autocomplete was a product that was out there, but ChatGPT still did not exist. We could see how these models were getting rapidly better, and we felt that AI could play a big role.

We also felt that, as a startup, if we went after these hard problems of allowing software developers to be productive in their codebases, we could differentiate, because it requires a lot of context understanding. I guess that’s something we’ll dig into more.

There are a lot of dimensions to the problem. Do you want to give me a little sense of how big a typical Augment customer is? You could measure that in employees, number of repositories, or lines of code. How big of organizations are you guys targeting?

We typically target organizations that have hundreds of developers. We do stretch higher, so we have some customers who have thousands of developers.

In terms of how many repositories, that really varies from customer to customer. Some customers use monorepos, like we do internally, and some customers use many different repositories, maybe one per microservice. That varies a lot.

In terms of lines of code, I think it starts probably with millions of lines, and then it goes up from there.

That highlights one immediate challenge right off the bat. When I do my own little projects, my default workflow, unless I’m testing something else, is that I’ll usually have the AI write a little script to print the entire codebase to a single file. Then, for a while at least, I can just copy that entire file, put it into the context window, and ask the AI for help.

I’ll take it to o1 Pro to do some planning, or Claude, or now I’ve got Gemini 2.5, which can take me farther, but it still maxes out at 1 million tokens, which obviously isn’t going to handle the whole codebase.

What do you feel isn’t done well, or is sort of missed, by—maybe take Copilot, if we want to pick on one, but you could say other offerings in the market? Maybe you could cash that out in terms of: What are the frustration points, or the places where you see developers just not getting the value that Augment can deliver?

We’ve all seen these one-shot examples: “Oh my God, it wrote this function for me,” and all that kind of stuff. But where does the conventional approach break down in practical terms?

From what we’ve seen, if we’re operating inside a codebase that doesn’t fit into the prompt—and today, even if we have 1 million tokens of context length—the ratio is roughly 10:1: 10 tokens per line of code. That only gets us to 100,000 lines of code, which in industry is still considered a small project.

There are other downsides, as the project grows, to actually putting all of that in the context, which we can talk about separately. But I’d say the problem there is that, when you’re working in a large codebase, you really have to keep in mind not just the work that you’re doing as a developer and what you’re focused on, but also the context.

That could be very obvious things, like: I need to call a few APIs, I need to call them correctly, and I need to put in the right parameters. I also need to call them in a way that respects the conventions.

Maybe there are multiple ways to call them. Maybe there are multiple ways to achieve the task that I'm trying to achieve, and we want to be respectful of the conventions that are in the codebase. These are all things that, if you're a developer inside an organization and you've worked there for a while, you're already familiar with—the right way to do it.

But when you ask an AI model to do it and you don't provide it with all of that context, it's going to struggle. Basically, it's going to give you bad predictions, whether it's a bad completion or a bad chat answer. Since at Augment we've prioritized context from the beginning, we have full codebase understanding built in by default to every feature.

If you're getting a completion, it's going to take the context into account, whether that means looking up the function you're calling or looking up other examples of its usage. If you're asking Chat the question, “Where is this function that I used 6 months ago and can't remember what it was?” it's going to search through your whole repository. With agents, we see that this actually matters even more because when you're trying to get these models to achieve more and more complicated tasks in a codebase, context becomes ever more important, simply because there's less supervision from the developer as the agent is working.

Yeah, context, as Tyler Cowen says, is that which is scarce. He said that before we got into the LLM era, but it feels like it's 10 times more applicable to the LLM agents that we're all trying to figure out how to make work for us than it ever was for humans.

I guess I'd love to dig into how you're making this work, because a couple of things have really stood out to me as I've studied the company and used the product a little bit. One is that the blog is outstanding. There's a lot of technical information shared on the blog, and that's an excellent resource for people to get a good sense of what you're doing.

A theme throughout many of those blog posts is really pushing hard on bringing a lot of resources to bear for an individual user. One way in which that manifests, as I understand it, is that, if I'm not mistaken, literally every keystroke that I make fires off a request to the server, which then begins to search the codebase to try to figure out what I'm doing right now, where I am, and assemble the useful context.

So maybe let's get into context management. You could take this in many different directions because I know there's a lot to it. From the second that I open the app—and we could also talk a little bit about how it's an extension of VS Code, not a fork, and there's a whole debate going on as to what's the right way to go to market—maybe you could take that one first if you want.

But when I open the thing and think, “Okay, here's my repository, and I'm new,” what's happening behind the scenes as it's indexing and getting me ready to put me and the app together in a position to really use a lot of compute at runtime?

We've actually explored several different approaches to codebase understanding. I think the approach we landed on was the third one that we tried, and each one of these was a multimonth research project to try to figure out whether we could make it work. What we landed on at the end could be described as RAG.

What happens behind the scenes is that we upload the code, we have our own custom-trained retriever models that we train for the purpose of codebase understanding, and then we index the code using these models. That's what happens when you open Augment and it says, “Indexing your codebase.”

Once that's done, on every keystroke and on every chat request, we send a request to the model. Part of processing that request is figuring out which parts of the codebase are most relevant to show to the model so it can make the best possible prediction for the user.

There's quite a bit of speed optimization that goes into making all of that fast. It's one thing to index a large codebase in the background, but it's a whole different story to say, “Okay, this completion request needs to finish within, let's say, on the order of 300 milliseconds.” That needs to account for both retrieving everything that's relevant from the codebase and actually making the language-model call to generate the completion.

We prioritize both quality, so that the retriever is good and, end to end, it actually feels like it understands your codebase, and speed, because to us, speed is a super important feature of the product.

You can obviously calibrate exactly how much you want to share about the details here. I recently did an episode with Andrew Lee of Shortwave, and they have a pretty similar approach. You sign up, and the first thing they do is ingest your entire Gmail history, which can be a lot. Then that goes into their database.

He's a database guru, and he told me—and I don't know if you'd feel the same way—“Well, yeah, we can pretty much tell all our secrets because by the time anyone figures out what we've done and tries to recreate it, we'll have a whole new generation.” I'm not sure if you feel quite as confident on that dimension and would be willing to share how things currently work, but to the degree that you can—long preface—I'd love to understand a little bit better how you're chunking code.

I think people have broadly come to frustration with RAG, and there are a number of different reasons for this. Sometimes you can fail at every step. How are you chunking? When you get a hit on a chunk, are you then expanding out to make sure you have the surrounding context that's needed, so it's not just that one function out of a broader class loaded in isolation?

There's context management, and then there's effective context management. I guess that's what I'm really trying to get at. How do you make it not just fast, but actually good, so it has the right information that it needs?

I probably can't be as open because I do believe that there's quite a bit of secret sauce in what we do. It is true that getting RAG to work well is very challenging, and in my experience so far, getting it to work well on code is even more challenging than in other domains.

To give an example of why that is, let's say I'm starting to type a piece of code, and let's say there's enough context there to understand what it is that I'm trying to do. Maybe there's a comment or something, although often there's not even that. My cursor is sitting there, I'm trying to get a prediction out of the model, and we need to know which pieces of code are relevant to make that completion.

This is a very different situation from a chat or question-answering system, where the user is asked to provide the context for the request. You start with an instruction or a question. You have a lot of context for understanding what it is that you're going to be looking for in your knowledge base.

With code, at least with chat, you have that. With completions, it's more passive. We're trying to both infer what the developer is trying to do and then figure out which code is relevant.

Let's say we—or the model—figured out that we need to call a function. Then the question becomes, “What pieces of code are most relevant to help the model make that function call correctly?” We could pull up the function signature. We could pull up example usages of that function. We could pull up other pieces of code that maybe serve as counterexamples.

Another thing about codebases is that they evolve over time, and we see a snapshot of the codebase. If we're pulling up examples, those examples could be new, or they could be obsolete and just left around in the codebase. The developer may not actually want to call them that way. So it's an extremely challenging problem.

What I can say is that we use a mix of different techniques. We use RAG, we use some amount of static analysis on code, and there are multiple models at play to provide the best possible context to the model. We also often let the user steer, because these systems are not perfect. We need a way for the user, especially in chat, to say, “Okay, I'm actually pointing at this directory,” or, “I'm pointing at this file.” This can also indirectly help steer the retriever.

There are multiple things at play. On chunking, there are definitely better and worse ways that you can do it, and it's true that code has more structure that you can hang on to. What I can say is that improvements in chunking are more to solve problems that are in the tail. Maybe I can say it like that. If you have strong retrievers and strong models, chunking shouldn't be a blocker. Yeah, I think I can say that.

So, yeah, one thing that jumps out to me there is how you started with the assumption that the user is typing code into an IDE in the traditional way. This may also tie back to the go-to-market as an extension of VS Code, as opposed to a fork.

I'm so AI-pilled myself and always trying to do 2 things at once. Usually, I'm trying to accomplish some goal in a project, but also learn about the latest AI capabilities or use Augment or whatever. I'm always looking for these two-for-ones, and I think that probably puts me in a very different pattern of behavior from what you typically see. As I've been using it over the last few days, I've done it entirely through the chat panel, and I basically don't really—almost never—get in and start typing functions myself anymore.

Where are people on that today, generally? What is the balance of approaches that you're seeing? And, by the way, I'm sort of a mid-programmer, which is maybe why I'm so drawn to the chat experience. For the pros, what's the balance between those who are working the old, traditional way, file by file, and getting this assistance proactively served up to them, versus those who are saying, “Okay, I want to interact with an AI and have it help me, but I'm going to give it an assignment in a sort of chat or agent-type paradigm”?

Yeah, there's definitely a distribution. I think when we were talking about completions and chat, we noticed that there do seem to be 2 camps of developers, and of course there's a lot of overlap. I doubt there are many people who only use completions or only use chat, but there are certainly developers with a preference for being a lot closer to the code, I would say, who don't use chat much but really love completions. Now, also, Next Edit is kind of a way to—okay, you get completions; they might be away from your cursor, and they can delete code and edit code, not just add code. But it fits in very nicely with the workflow of developers who want to keep their focus on the code.

Then we see a lot of developers who really only use chat. That is fairly common. The thing that's changing now is that, as we're building agent mode, you can take another step away from the code and really let the model edit multiple files and run your tests. You're taking another step away and supervising everything, and then you can dig into the code when needed.

This is something we've seen: when you work on a large codebase, you pretty often have to go back to looking at the code and making some changes yourself. That's pretty frequent, which is quite different from the zero-to-one experience that I think we talked about before. So, I would say that for large codebases, as far as I can tell, most developers are comfortable being in chat a lot of the time and using completions and Next Edit. The switch, or the move, to a fully autonomous agentic flow will take longer. I think agents and models will need to improve before that becomes the default mode for enterprise developers. Let's say it will take longer, but I feel like that's the direction we're going in.

Yeah, certainly. This can get into almost ideological territory very quickly, but I'm sure for you it's much more a practical question around extending VS Code or doing a fork. Is that a matter of meeting developers where they're comfortable and not asking them to change too much, or are there other big decision drivers that have you in the extension paradigm?

Yes, I think it starts from meeting developers where they are. We have a VS Code extension, a JetBrains extension, and Vim support. This is really about not wanting to force developers to change how they work.

I think with the forks, these are all VS Code forks because VS Code is open source. You can say that if you switch from VS Code to a fork, you're not changing your workflow that much. But if you're asking a JetBrains developer to switch to a VS Code fork, that's a pretty substantial change to their workflow. That's one consideration.

I would say there are also other considerations with the fork. Doing a fork means you need to keep up with updates, especially security patches, which becomes extra maintenance work that you have to do. Especially if you're selling to enterprise, these security considerations can matter.

Now, the downside of not having a fork is that there are certain UI things that are harder to do or sometimes impossible to do. Although I have to say that with the VS Code API, we've been able to do a lot within VS Code. I don't think this has been a very substantial limitation. Sometimes we've had to work harder because we can't just go and change the VS Code itself.

This is another place where I suspect that the more we move to agentic flows, the less we have to do inside the text editor. Once you're building an agent inside VS Code, you have a lot of freedom in what to do because you can open panels, put webviews in there, and have full control over what's happening. So, my sense is that this distinction is going to become probably less important over time, but I can't promise we won't do a fork at some point. There's certainly a trade-off there.

Yeah, that's interesting. The point about security and being able to piggyback on all the hard work that Microsoft has already done to establish trust definitely makes a lot of sense. I've had enough experience with the security review processes at enterprise customers—not nearly as much as you've had, but enough to know that it's not where I want to be spending my time. To the degree that you can shorten that process, it certainly has a lot of appeal.

Yeah, exactly.

So, going back to retrieval—and again, you can calibrate your answers however you want—but for practical guidance for other people building their own RAG apps, do you have a favorite vector database?

So, we actually built our own vector database. I can explain why we did that. There was nothing out there that we found at the time that addressed all our requirements.

What do we want the user experience to be? We want the user to feel like the model understands their whole codebase, and we want it to feel like it understands the current state of the codebase. So, if I just wrote a function in a file, or had chat write it for me, and now I ask chat, “Okay, implement the tests,” or I go to a test file and start typing a completion or a test, we want the model to understand that this is something I recently did and have it all indexed and available.

That means giving every developer—or giving the model—a real-time view of every developer's codebase. It has to be real-time, or feel like real-time, and it also has to be different for every developer, because if I'm a developer on a team, I work on my feature branch and you work on your feature branch. We cannot have those things mix. That's also a security requirement.

In terms of a vector database, you need something that allows almost real-time updates to the index, which is already a significant requirement for a vector database.

And it also needs to be able to have queries based on different views, right? I have a slightly different set of files that I'm retrieving from than you, but we still want to deduplicate. We still want to have one database that captures our repository, and not deduplicate that for every user on a team. We did not know of a product that did all of that.

And there's a technical reason for it. Typically, the way vector databases work, when you query, it's pretty expensive to do a full query every time, and so you apply some kind of statistical algorithm. Maybe you cluster your embeddings and search in the cluster. There are all kinds of ways to do that.

Taking that kind of approach, or one of these standard approaches, means that it doesn't work well both with indexing or updated indexing, because updating the clusters can be expensive. Also, views or queries based on views are hard, because if you're doing a statistical query and you only have a subset of the files that you're retrieving from, you might miss them completely when you're doing a statistical query.

And so it was certainly a difficult engineering problem to build a vector database for us. We still keep iterating on it, especially as we have customers that have larger and larger codebases. We need to keep scaling up solutions. So there's a project ongoing right now addressing scale requirements with repositories for us. But, yeah, we ended up building our own.

It's fascinating. I guess you started in 2022. Is that right? Yes. Yeah. It may or may not be different today. I wonder, this sometimes strikes people as crazy when I float ideas like this, but that almost sounds like a product unto itself. Have you thought about that?

It's come up. The thing is, when you build an AI lab and then an AI product on top of that, you run into many things that can become a product on their own. One of the challenges is to stay focused and have a concrete vision of what we're trying to accomplish.

So this comes up as, “Oh, this could be a product.” This comes up as, “Oh, should we fork?” And then, well, who are our users really? Who are we catering to? Are they going to want to fork, or are they going to want to prefer their IDE? There are questions like this that come up all the time, especially with something like AI, where it's a completely new technology that keeps improving rapidly, and you have to keep up with what's happening and make the right bets.

Short story: yes, I think this could be a product, but we're trying to stay focused on building the best AI assistant we can for developers.

Are there any—you probably haven't kept up with the evolution of other vector databases—but for people who are trying to pick one, so many people right now are at the stage of either embarking on a sort of RAG app for their business, probably for internal use, or maybe they've made one and it's not quite working well enough, and they want to take the next step with it. Are there any general guidelines that you would give people for how to make this part of the system work?

One that I have in mind—it sounds like you kind of have a version of it—is that I personally would almost always insist on some sort of hybrid, like structured query plus vector. At the beginning of this RAG wave, people were just doing pure vector search, and that seemed to be kind of a mess. Having some ability to do a classic SQL-style WHERE clause along with the vector similarity, whatever, seems important to me. But I wonder what your lessons or guidance for the masses would be based on all this experience.

Yeah. I would treat this as a research problem and start with some off-the-shelf vector database, unless there are pretty clear engineering requirements that would preclude that. We have, I'd say, pretty special requirements, because it all has to be low latency and so on. I don't think most RAG implementations need all of that, and so I would probably start with some off-the-shelf vector database.

I would focus more on the quality. For the quality, one thing that's pretty important is to have an evaluation dataset that you trust. It doesn't have to be a huge dataset. You can start with even 10 to 20 samples labeled by hand. That's how we start most projects.

Actually, most research projects will start with collecting 10 to 20 samples labeled by hand and then start with some baseline. Take an off-the-shelf retriever—whatever is easiest to use—run it on your evaluation, and get a baseline of how we're doing. Are we solving 20% of samples? Are we solving 80% of samples? Probably it's going to be somewhere in between. Is that good enough?

Start iterating from there and hill-climbing on your evaluation dataset. When the evaluation gets saturated—basically, when you've managed to solve it—expand it. Add more samples, make them more diverse, and make them harder. I think coming up with good evaluations and being diligent about running those evaluations is, in some sense, one of the hardest things to do in research—not because the work is so hard, but because it can be pretty tedious. This is the way to get to good results.

Starting with that, things become straightforward in terms of, “Should we just do vector? Should we do structured queries?” Well, let's try it. Everything becomes an experiment. Let's try it on the eval set, and the eval set will tell us, because we're basically reducing the problem to hill-climbing on an eval set. That is the ideal situation.

I will say, certainly, doing pure vector—let's say it like this: real-world retrieval systems are almost never a single thing. It's almost never, “I'll just do embeddings, and I'll work on the embeddings really hard, and I'll get the best embeddings, and they will solve the problem.” That almost never happens. It's usually a mix of different techniques. Vector and structured search could be combined with any other signal that you can bring to bear on the problem.

The models today are good enough that you can actually throw a lot at them in the context, and they will deal with it. In some sense, recall becomes more important than precision. You want to make sure that the right chunks are in the context. That's really, today, with modern models, the thing to prioritize.

Now, I described the ideal situation where you can come up with a dataset that you trust, and all you're doing is hill-climbing on that dataset. It's very important to start like that, but at some point, necessarily, your evaluation dataset is going to diverge from optimal user experience.

I've never seen it otherwise. This has happened in every one of the projects that I think we've done. It's really hard to capture user experience and map that to one number. Not just because there are multiple axes, but also because we don't really know how users use AI products. There's a whole distribution of what they put into the prompt box, and there's a whole distribution of what they expect to get out. You can't really boil it down to a number. And so once you have something, dogfooding is crucial for understanding where you are.

And then once you have users, user feedback is crucial. You have to take all these things into account. I would recommend starting with an eval set, but then understanding that you also need these other sources of feedback to iterate. That, I think, is a quick summary of best practices for how to go about this. Once you're able to reduce all these questions—basically, the place you want to get to is: can you turn all these questions of what to do from philosophical questions into experimental questions that you go test? Then that's when you can iterate and really move fast.

Yeah, that's great. It is striking to me. For whatever reason, I find myself doing more projects where there isn't quite a ground truth that's so easy to hill-climb on. With my company, Waymark, we do video creation for small businesses, and there isn't a single answer to what the right or best script or voiceover script, or selection of images, is for a particular small business. There are definitely better and worse options, and sometimes it's very obvious; other times, it's the subject of disagreement.

We've definitely had plenty of cases where we ask 2 different people, and they see that one is better than the other. It's not unanimous in most cases, but it is definitely enviable, from that perspective, to have these almost irreducibly vibey tasks. The idea of being able to just climb a hill is quite attractive.

But in both cases, I do think it's really important for people to keep in mind that you can start, and you should start, with a pretty modest-sized data set. I have a whole presentation that's much more about the social side than the technical side of getting your team on the same page about what 10 instances of a task that are really well done look like. It's amazing to me how often that ends up becoming the stumbling point.

I think it is often because of what you said: that's just tedious, and they also don't have any chain of thought, which is sometimes really helpful if you want to do a supervised whatever. I'm on the verge of the soapbox. But, yeah, 10 examples will take you far. If it's objective, great. Even if it's just a vibe task and you're demonstrating what a job well done looks like, 10 examples is the first place to get to. From there, the world can open up a lot more.

Yeah, just to add to that, 100%. The big advantage of having these few examples before you go to hundreds, if you can, is that you become very familiar with them. You can hold them all in your head, and so the labels—the ground truth—is less important. I'd say you can still hill-climb on 10 samples even if the evaluation procedure is completely manual.

I just trained a new model, or I have a new RAG setup. I will run it through the 10 samples. I will run the before model and the after model, and I will compare them by hand. I don't have to boil it down to a number, but I can also go based on vibe. So I agree with you that the real minimal thing to start with is 10 samples. The number can come later. I totally agree with that.

One other thing you said that I thought was worth re-emphasizing, too, is that the key thing is to make sure the model has what it needs. Worry less about other considerations, like distracting it with wrong information. This is, of course, evolving quickly, because that used to be a much bigger problem not too long ago. As you said, modern models—the word “modern” is important—are what everybody should be using, but our expectations aren't necessarily always keeping up with what the latest models can do.

One way I generalize that for people is: turn your hyperparameters up. In general, you typically have a choice in a lot of these RAG-type setups of how many chunks you're going to take. Are you going to take the top N chunks? Or, if you're going to expand out from a chunk, how much should you expand out?

From what I see developers doing, they're leaving all those settings too low. The right thing to do is usually turn them up. Yes, that might make it slightly slower. It will make it a little more expensive, but I don't know if I can think of any exception where turning those things up didn't more than pay for itself, even with those marginal cost increases, in the sense of the time savings you get from getting to something that's working better.

So I don't know if you have any exceptions you would put on the “turn your hyperparameters up” rule of thumb. From the blog, it does seem like you guys are definitely asking, “How can we use all these things to roughly the maximum?” But I'm interested in your take on any nuances you would add to my simple rule.

No, I fully subscribe to that rule. You just have to be aware of the trade-off between latency, cost, and quality. It's really as simple as that. If you're okay with the extra latency and cost in a RAG context, it's just better to show more, because these days the models have been trained to deal with it.

It didn't used to be the case—definitely not 2 years ago, maybe not even 1 year ago. I'm not sure, but roughly around that time is when models got the RAG training to be able to deal with a lot of distracting information. And so that scales really well. I expect we'll just continue scaling, because the attention mechanism in transformers is basically built to do that. It's built to sift through all the noise and focus on the relevant parts. With sufficient training, it makes sense that it will work.

I think I've also seen this bias toward putting less in there, especially if you're coming from a background of using models the way they were 2 years ago or before. You had to be a lot more careful with your tokens, but that has changed. If we're focused on RAG, then, yeah, I think that's just the right answer, and it also certainly makes the research much easier, because improving the recall—that is, improving the ability of the model to find the right chunk within the first 50 or 100 or something like that—is doable.

Improving the recall if you have fewer chunks—if it needs to land in the top 10 or something, or top 5—that task becomes exponentially harder, basically, the less context you have. So if you can give it the room to do it, then the research task becomes much easier.

I can say the place where adding more context doesn't seem to scale yet is with instructions. Giving the model tons and tons of instructions that you expect it to follow, in my experience so far, doesn't scale that well. It will start ignoring instructions if there's too much in there, skipping steps you asked it to do, and things like that. But that's not a RAG problem. That's just a different kind of prompt-scaling problem where models are not yet good enough.

Yeah. Okay, that's a good point. You mentioned training. You've trained your own retriever models. This is something that I think—I don't want to bias your answer too much—but my sense is that a lot of software engineers and AI engineers are attracted to the notion of, “Well, of course we're not going to pretrain from scratch, but we'll grab some off-the-shelf thing and customize it for our own purposes.” I wonder what guidance you would give people on when that is, in fact, a good idea.

I once made the bell-curve meme of what's genius, what's dumb, and what's in the middle. My thing was that, on the extremes, you should just use OpenAI embeddings, and in the middle was, “Oh, well, we'll do this complicated thing. We'll fine-tune our own blah, blah, blah, blah, blah.” It's worth it for some. You're in that situation where you have a lot of resources and a very ambitious project, but where do you think it starts to become worth it to take on that sort of challenge versus just using the best off-the-shelf thing you can find?

Yeah, that is a very task-dependent question. Anytime you do research, I think there is just a general human tendency to reach for complicated solutions too quickly. My recommendation would be to try hard to bias toward simplicity. The simpler, the better, and rely on your evaluation—either vibes-based evaluation or numerical evaluation—to guide you.

I'd say, again, it's hard to give general answers, because the answers are going to be task-specific. So I'd rely more on establishing a solid process for finding the right answers for your particular use case. That starts with an evaluation set that you're comfortable with, that you mostly trust, and that you can run through different iterations of your model or your system to test.

The other thing that I think is important to optimize for is iteration time. The faster you can run experiments, the more likely it is you'll find something that's good enough or something that's better than what you currently have. So experimental iteration time is something that's very much worth thinking about.

Taking experimental time down from hours to minutes can have a lot of impact on not just how fast you get to a solution, but also whether you even get to a solution. Cranking through 100 experiments versus cranking through 10 experiments, the chances of you finding the right thing in those 100 experiments is just much higher.

It's kind of like the RAG problem. What's your chance of finding the right chunk in the top 100 versus the top 10? It's just much higher. It's also like that with experiments. So if you can afford to run 100 experiments, then you're going to try 100 different simple things.

Maybe you're going to try SQL-style queries. Maybe you're going to try 5 different open-source models, and maybe you'll try OpenAI embeddings plus other things. You'll find that one of them, for some reason that was really hard to predict, actually works better for your use case. So I'd prioritize experimental iteration time and being able to actually trust the result of an experiment, with an eval set to tell you the answer.

And once you've tried some simple things and nothing seems to work, I think another thing that process gives you is a kind of feel for, “It looks like nothing out there is really doing what I want. Maybe I should start thinking about fine-tuning an open-source model.” Or, “Okay, this isn't exactly there, but it's kind of close. It's not that far. Probably, if I keep going this way, I will be able to make it good enough without fine-tuning.”

That's the sort of information you get by doing a lot of experiments. So my suggestion would be to do that and then let the experiments tell you which way you need to go and when.

Okay, cool. That's great. There is so much that I wanted to cover, and I don't think we're going to get to all of it. So I'm going to have to start to pick and choose, and then I'll refer folks to the blog for some deeper dives on stuff we don't get to.

But one thing I definitely want to cover is reinforcement learning from developer behaviors. Obviously, reinforcement learning on language models in general is having a moment. The floor is yours. Tell us about reinforcement learning from developer behaviors.

One advantage that we have as a company that both does research and builds a product for users is that we are very close to our users. We get feedback from them on Slack and Discord, and they also send us their data.

Now, for enterprise customers, we do not look at that data. Of course, everything is audited and behind access controls and so on. But we do have a free community tier that's for anyone who wants to use Augment. It could be on open source; it doesn't have to be on open source. Anyone who's comfortable with us looking at their data and also using it for improving our own models can use it. So there's a very clear separation between those two things.

But on our free tier, certainly, we find value from that data because one of the things that's universally challenging about building AI products is that we don't really know the, let's call it, input distribution or task distribution. What do users want to do? How are they trying to use the product? What do they expect to get out of it? By collecting this data from the free tier, we get a glimpse into that.

And in fact, for coding, we get more than a glimpse. That's one of the nice things about coding: it's quite different from a chat interface, right? In a chat interface, the user asks a question or assigns some task, gives an instruction, and gets an answer. They can continue steering, but we don't know what the ground-truth answer was. By comparison, if they're working on code in their IDE and we follow what's happening in the IDE, we eventually know what they were trying to do, because this is where they actually work.

The way this connects to reinforcement learning is that the idea with reinforcement learning is that you're not just training the model by showing it examples of what to do. You're actually showing it contrasting examples. Every sample contains an input, and then it contains a better and a worse output. The model learns from that contrast to do better over time.

This is a very powerful paradigm because it means that it's not just that there are correct and incorrect answers. There are better and worse answers. You see that in coding just like anywhere else. The answer could be correct in the sense that the algorithm is correct, but it could have the wrong style. Maybe that's not what the developer prefers, or maybe the style does not align with the rest of their codebase.

There are actually multiple axes on which a sample can be better or worse, and that's the kind of signal that reinforcement learning tries to capture. We've applied that technique initially to the completions feature. That was kind of our first reinforcement learning project, where we used examples from the model and what we know from the user in order to improve the model and align it better with what users expect through reinforcement learning.

We ended up with a better completion model because of that. I mentioned that we train retrievers, but we also train the generation models that we use for completion and next edit.

Yeah, there's a lot that I'm interested in digging into more deeply there. I guess, for one thing, have you benefited from DeepSeek and other recent algorithm releases? I think it's safe to say that GRPO broadly has blown a lot of people's minds. Did it blow your minds, or did you feel like you already had a pretty good read on what was working such that it wasn't such a revelation for you?

The algorithm itself, I don't think, was—I'd say there were known problems with existing RL algorithms that every subsequent iteration addressed. So I don't think GRPO for us was a revelation. It was more of an iterative improvement.

I think the DeepSeek work was remarkable. They actually implemented chain-of-thought reasoning training. It was super impressive work and a very nice paper. I wish they shared more details on how they did it, but I still enjoyed reading the paper.

We do benefit a lot—not from that particular work, but I can say that we don't train models from scratch. We made a bet very early on that open-source models would rapidly become better. That was at a time when open-source models were really not good. This was long before Llama.

I previously worked at Google and worked on some of the training of large models. It was clear that this is something open source could do, because at least at the level of training base models, and now also beyond base models—actual instruction-tuned models—there is almost a playbook for how to do it.

If you have the resources and the people who know the basic techniques, which are for the large part in the literature—you can read papers and learn how to do it—then you can train very good models. So we made that bet early on. We don't train from scratch. We do a lot of post-training on models for retrieval and generation, and we definitely benefit from open-source models coming out. We generally try to keep up by basing our models on the best available open-source model that's out there.

This particularly caught my interest, this reinforcement learning from developer behaviors, because I've been looking for something like this to emerge for a while. My sense is that the compute requirements for reinforcement learning aren't so crazy, and the datasets don't even have to be so huge. It seems like a lot of product user bases, or just communities in general, if they're passionate about a certain subject or whatever, could gather enough feedback or behaviors from people to power this sort of thing.

I haven't seen too much of it, and I was wondering why it wasn't happening. Now it seems like it is happening. I wondered, though, if you could shed any light on where you think it's going in particular.

One might think that the fact that there's not a true, absolutely canonically right answer, as there is in a math problem with a numerical answer, would suggest that maybe this process would top out at human level and might not go past human level. How would it go past human level if it's learning from humans?

The flip side of that also would be that it would seem like this approach would be very extensible to reinforcement learning from lawyer behaviors, or reinforcement learning from doctor behaviors, and other—basically anything where you can gather enough data that's pretty trusted, even if it's not absolute bedrock ground truth.

What do you think? Is there a top-out that we should be thinking about, and is there any sort of limit on the breadth of how far these sorts of approaches could generalize?

I guess I have a few thoughts. First, if we think about the trends we've seen in the beginning—start with GPT-2 and then GPT-3 and the scaling loss trend—what was the trend? There was a whole lot of data out there on the internet. Let's get as much of it as we can, process it properly, clean it up, and filter it, because there's also a lot of garbage out there. But basically, that was the first resource that large language model training reached for.

I think at this point that data resource is more or less exhausted. So what can we do? There are roughly three things we can do, let's say. One is synthetic data. We know that we can generate more data out of these models to train new models, so that's certainly one approach.

Another approach is to pay contractors to give us the data that we need. That's how most RLHF works. That's how you train something like ChatGPT, essentially, if you don't have any other data sources.

And the third is user data. If you have real user data, if you can figure out how to use data from users who are using your product to do real work, that's in some sense the holy grail, because that is the closest you're ever going to get to the actual distribution of what users are trying to do. It is what they're actually trying to do.

So there's no distribution gap in that case, if you can do it well, between the data you're training on and the data you're going to encounter in the wild at test time. I think the reason we haven't seen more of that is that there aren't that many products that are amenable to it.

So I think if you imagine beyond coding—doctors, lawyers—what do they do where you can actually get the ground truth? If they're editing a document, you can get the ground truth just like you can from an IDE. If they're using a chat interface, it's a lot harder to get the ground truth. Maybe you can guess at the ground truth because maybe they tell you, "No, no, no, that's not what I meant. Do this or that." And then you can, but there's a lot more work that you have to do to extract the ground truth from something like that.

I do believe that as we exhaust the available information on the internet, user data is just going to become a lot more valuable, and people will pay more attention to it.

In terms of tapping out, I would just say that everything we've done so far has been training models based on human data. There's nothing really new here. All the data from the internet is human-generated data. RLHF is human-generated data. We can automate some of it, so we can throw models in there and let humans supervise at a higher level, but human supervision is always there so far. In the future, if we want to break away from that, we need some other source of signal. We need some other source of reward for these models.

That's where I think code is probably the place where it's going to come first, because the thing that's special about code is that you can execute it and get feedback from that. So I can see how, in the future, we'll be able to do that, and some of it is still happening. If you look at the way DeepSeek was trained, they don't say a lot about how they did it, but they do get feedback from code execution for RL purposes. It's a very natural fit.

For code specifically, I think we'll be able to do a lot more of that. For other domains, we'd have to find something else. If you're asking the model to write a story or a poem or an essay, how are we going to automatically assign a reward to that if we don't already have a better model that can judge what this model did? Right? So that's where I can't think of a way to go beyond human capabilities there.

But when you do have a ground truth that's separate from humans—code execution, maybe for science this could be experimental validation, things like that—then we'll be able to, at some point, shift away from humans and rely on these other reward signals.

Have you seen any reward hacking in your reinforcement learning from developer behaviors?

In our reinforcement learning? No, I can't think of an example. There was nothing as spicy as that. The mistakes were just not understanding what the user wanted; that was, I think, the most common mistake.

Okay, we'll keep an eye out. We're all looking for reward hacking. We should be these days, I think. So let's see—just prioritizing, kind of triaging a little bit. Maybe shorter answers, if that helps.

No, you're doing great.

Maybe a minute on just the economics of businesses like this. It's public, and again, you can go into as much depth as you want. It's public information that you guys have raised $250-ish million. I looked on LinkedIn; I saw 100 employees. Perhaps not everybody's listed there.

But if I were to just do traditional SaaS app math and take a multiplier of employees times some Bay Area salary and then try to calculate a runway, I get to a really long runway. And that's before any revenue, and it sounds like there's quite a bit of revenue. So what are you doing with the money? Are you burning a lot on training models? It sounds like if you're not pre-training, that doesn't seem like it would be. Are you subsidizing users? You are subsidizing open source.

So, yeah, maybe to the degree you can, I'd be really interested to hear about the economics. And then a slight extension of that would be: Is there a 10× more expensive version of the product that you could dream of or imagine? What might that look like?

Yeah, it's a great question. One thing we've learned is that AI is different from the SaaS businesses we're all used to thinking about. You're developing in your basement, you set up a Google Cloud project, you start serving users, and it's all very—it's pretty cheap, I would say. Your main cost is salaries and so on. But AI is more capital-intensive. There is the training, but inference is also very expensive. Serving all those requests at every keystroke, then the chat requests, and now the agents—it gets expensive quickly.

There's some amount of subsidizing users, like in the free plan, and I think everyone in the space is trying to figure out the economic model right now. On the one hand, usage is exploding, and I think it's going to accelerate dramatically. Models are getting cheaper, but they're not getting cheap. It's not matching the pace at which usage is growing. A given model is getting cheaper, but people always want to be on the latest model, and that's not getting cheaper as quickly.

All these factors combined mean that running an AI company, both if you're training your own models and if you're not, can get pretty capital-intensive. That's the short answer.

Now, on a more expensive product, if you look at the shift, we're launching our agent feature. The cost of agents compared to chat, for example, is a substantial jump, because with agents you give it one instruction and then it goes. That will probably generate 10 or more language-model calls, including large calls for editing files and running whole commands, then parsing their outputs and doing all those things, all from a single user instruction. And that's at a point where we are giving users a single agent in their IDE. So it's just going from chat to one agent.

On the other hand, the value is clearly there. I can say personally, I have not written a line of code in several months. I've been using our agent; it has written a lot of code. I personally have not had to. The value is very obvious with these things, and it's super early.

I expect usage of agents to explode over the next year and, with it, the cost. If you're talking about a large jump in cost going from chat, let's say, to agents, I expect there to be a jump that's at least as high once we're able to unlock the full value out of agents. I don't think that cost decreases in models are going to keep up with that. So cost does become a challenge. The whole thing is just very capital-intensive, and cost is actually a major factor, unlike traditional SaaS businesses, I think, where it's not as much of a factor.

Are you managing your own clusters? Are you actually buying all the GPUs and managing them in-house, or are you leasing or renting from someone else?

No, we're leasing. We're not in the business of managing data centers. We're leasing the GPUs.

So does that translate to a higher price point at some point in the future? My rule of thumb has been that I think companies should expect to spend $1,000 a month on AI to augment their employees—no pun intended—in the not-too-distant future.

I personally am probably halfway there just with stuff that I've signed up for. Your prices are $30 and $60 a month, and it feels like, if I'm right, you should probably be 5× to 10× those prices. But I don't know. Is that where you think it goes or not?

I don't have definitive answers. To me, these are open questions. There's even a question of whether it should be a fixed subscription price or more of a consumption model.

Our current pricing model is somewhat consumption-based. We sell credits, and if a developer uses the product for a given month, they consume a credit; if they don't, they don't consume a credit. That's different from the more common seat-based pricing, where you just sell seats and pay no matter whether users use it or not.

So we already took a step in this direction of consumption-based pricing, which was meant to really align our interests with those of the users: you use it, you pay; you don't use it, you don't pay. I expect, because of the cost, we're probably going to lean—my guess is more heavily—into that model, but I'm not sure yet.

So this question of whether it's going to be $1,000 a month—maybe we'll end up there—or maybe we'll end up with a different model that's more aligned with how users actually use it. What I can tell you is that there's a very wide distribution in how users use these things. There are absolutely users who will justify a $1,000-a-month price point even today, and then there are users who don't. They just don't use it as much. So I think we and everybody else are trying to figure that out.

Let me throw another thing in there. I think right now we're all just thinking about user-driven agents, let's call them, or interactive agents, where the developer is kind of there. Maybe they go get a coffee and come back because it takes the agent a few minutes, but it's a few minutes, and the developer is kind of staying up to speed with what the agent is doing.

I think things are going to evolve rapidly over the next year, and it's not even clear to me if that will continue to be the dominant use case.

I'm pretty sure we will have agents that run for hours, overnight, or over days to accomplish tasks. I'm pretty sure we will have agents that work on non-user triggers. Maybe it's API calls, or maybe it's an agent that goes and does code reviews for you and things like that that are not just run automatically. If you're in that world, then you're not even talking about per-developer pricing exactly anymore, right? You're kind of putting intelligence into a lot of tasks that are not triggered by the user, or maybe there's a wide variance in the cost of what the user triggered.

I think the pricing model is going to have to adjust to that, at least in the short term, until all this stuff becomes super cheap. So, yeah, it's a complicated question, and I don't have a good answer for it. All I can say is, it's a good and complicated question that we're definitely thinking about.

Yeah, I think it feels to me like aligning interests with users is hard to go wrong when you're generally keeping that in mind as a true north. The thing that I am always allergic to is when I feel like the product is not performing as well as it could for me because I've got some fixed price, and they're trying to keep my cost to them under that price to maintain a margin. And it sounds like you're not doing that by basically accepting the fact that you'll have some $1,000-a-month-cost users and figuring you'll figure that all out later.

But, yeah, I definitely want to be that $1,000-a-month user, even if I do have to pay for it. What frustrates me is when I can't be, because I'm locked into a more conventional price point. So, okay, time is short. Maybe 2 more questions if we can fit them both in.

One is that you guys have a blog post on why you think RAG. It's multiple predictions, but the one that jumped out most to me is why you think RAG will trump fine-tuning. And here I wanted to just super quickly sketch an idea that I've been chewing on for what the drop-in knowledge worker of the future might look like and kind of get your reaction to it. Then the last one is just the future of the software industry: What should junior developers do as well?

So, drop-in knowledge worker. We've covered the RAG stuff: What's hard for the models today? They don't have context, and I always feel bad for them in some ways, because when I'm searching through my Gmail or my Drive or my codebase, one huge advantage I have is that I kind of know what's in there, and I know when I've found it. In contrast, the models today just get what they get, and they sort of have to do the best with whatever is returned, right? You can turn up hyperparameters, and that helps, but they don't know in general, “Have I found the right thing? Should I keep searching?”

And so I have this sense that continued pre-training is maybe one way to describe it on a company's proprietary data. Basically, try to get to the point where the model knows the company from an inside perspective as well as the models today kind of know the world at large, and then continue with your post-training, your behavioral refinements. But try to get to a point where the model knows, “Yes, I actually found what I'm looking for. This is the ground truth that I needed to go on this task,” or not, and therefore I'm going to keep searching and maybe using different tools until I get there.

What do you think? Does that seem too far-fetched, or how would you generally react to that vision of continued pre-training, so I know, “Yes, I found it,” or maybe that is not necessary for some other reason that I don't see?

Yeah. So, I can say there are a few challenges with that approach. One challenge with continued training is that, even though training is very sample-inefficient in that you need a lot of data for the model to learn something, typical company knowledge bases are too big to put in the context window, but they are not large in the sense of training data sets. They're actually typically pretty small. So, if you think of a typical code repository, it's not a lot of data to train on.

If you want the model to pick up on what's there, you're going to need to do multiple epochs, probably train on it multiple times, but then you quickly overfit, which you also don't want to do. You don't want it to just memorize what's in there. You want it to actually learn from it. So, I'd say one challenge is that the amount of data for this to be effective is typically too small.

Another challenge is keeping it up to date. You can work hard to make your RAG solution instantaneous, or you can reduce the delay as much as you want. Training models has more friction there. If you want to do it for email, for example, every user has their own email store. So, if you're going to be training a model for every user and keeping it up to date, there's not that much data, and the logistics of doing that separately for every user are tricky.

I think it used to be like, do you do RAG or do you do this? These days, I would honestly try to solve this problem with an agent who tries several approaches until it thinks it finds the answer. It doesn't have to be just, “We'll do retrieval, and did it get it or not?” You can do a lot more now, and so we've actually built a lot of that into the product.

The more advanced versions are not just a one-shot retriever. We do more to give you the best retrieval quality we can. In the interest of simplicity, models are so good now that I would reach for solutions like that and try them definitely before doing fine-tuning.

Yeah, interesting. So, you think you can basically, in short, get good enough performance without ever having a model that really knows, in a confident, intuitive sense like I do, that it actually has found the right thing?

Yes, I think so. I'm saying that based on the evidence we're seeing, at least as I interpret it.

Okay, cool. Last one, and it's not necessarily an easy one. We've got people talking about superhuman coders within this calendar year or into next year. I'm referring to, obviously, Dario Amodei, who said that repeatedly recently. I guess I wonder: Do you buy that that soon, or even if you extend the timeline a little bit? And if so, what do you think that means, or what advice would you give to people who are especially early in their software career today?

From what I see on the internet forums, it seems like people who are just coming out of school with a CS degree are sort of like, “Yikes, this is not what I thought I signed up for.” Not everybody can just pivot into being a machine-learning all-star. That's a great option if it's open to you. But for the rest who are like, “I did this because I thought I was going to have a nice, stable career with a solid income and never have to worry about my employment status,” where do you think we're going, and what advice would you give them for navigating the challenges that might be coming for them soon?

Yeah. So, first, I think it's good to separate the short term from the longer term. In terms of what Dario said, the way I understood it is: If you look at what actually happens in the near term, you go to a new line of code and ask, “What actually generated this line of code I'm looking at?” His statement very likely is going to be that it was a model or an agent that did that, rather than a developer.

I buy into that. Maybe it's not in the 3-to-6-month timeframe he mentioned. That's probably too quick based on the adoption that we're seeing, but not 3 years. I think it's shorter than that.

However, that doesn't mean that the model decided on its own what to do. It doesn't mean that it supervised its own output. It doesn't mean that it's fully autonomous. I expect that for a long time there's still going to be a developer there steering the model, and I expect that because this is how I work.

This is how I see people who are picking up agents work. You look at their code: Once they pick it up, the code was almost entirely generated by an agent. But if you took away the human, nothing good would happen. You wouldn't get anything useful out of it, because the models are nowhere near that good. They're not even good enough to say, “Here's the product requirements; go build this.” We're definitely not there yet.

So, I think I buy into that statement, but that doesn't mean we don't need software developers in the next year. In terms of advice, I have 2 kids; they're 7 and 14, and we're having discussions with the older one about what to go for. My advice is to go for a career that's more tied to the physical world.

It could be mechanical engineering or robotics or something like that, where it feels like it will take longer to be disrupted, because it's very hard for me to predict what software development is going to look like in, let's say, 3, 4, 5, or 6 years. That's very hard with the rate things are changing. I don't know where it's going to land.

I think we'll still need developers who understand the system, because if you're just vibe-coding your enterprise software, you will run into trouble. I can already see it happening with the code that I'm writing. It will get better, but I don't think it's going to get better at that scale. But then the question is, well, how many developers do we need, and how much software do we need to write? And I don't know.

So those are the discussions we're having with my older one. With my younger one, we have a bit more time to figure this out. Maybe at that point we have AGI and everybody can just do art. I don't know. But I'm glad we have a bit more time to figure it out with him.

Yeah. Yeah. Well, the local artisanal economy could be a beautiful future, as long as everybody has their basic needs met. How does this get operationalized for you in terms of your hiring? Are you hiring junior developers at all? Is there any on-ramp for somebody out of a CS program to get into a frontier company like yours?

Yes, there is. For sure. We look for excellence. We hire junior and senior developers. I think this is still a time where there's going to be a learning curve in knowing how to extract the value out of these models. Even if you're using agents, getting value out of them takes time, especially in an enterprise environment or in a codebase like ours.

But we're small—I wouldn't call us an enterprise—but even in our codebase, which is, let's say, small to medium size, using agents to navigate that codebase requires some skill. So I think for a while there's going to be ramp-up time, where it seems that, as always, people with less experience are also quick to jump on new technologies. I think we're going to see a lot of that in the near future, but the short answer is that we certainly still hire junior developers.

Great. Anything else you want to leave folks with before we break?

Uh so augment is is out. It's really good at understanding your codebase. So, I encourage you to download it, give it a try and really feel the power of an excellent AI assistant that that fits into how you work.

And I would definitely encourage people also to check out the blog for a bunch of deep dives. We didn't even get into the inference optimization work and all the detailed analysis of batch sizes, which I did think was super interesting. And there's a great write up of the next edit feature as well. So, there's plenty more to be unpacked from the augment team than we've had time for today. But nevertheless, this has been a great conversation. I really appreciate it. For now, Guy Gur-Ari, co-founder and chief scientist at Augment, thank you for being part of The Cognitive Revolution.

Thank you so much. This was a lot of fun. Thank you.

Thank you.