Brex’s AI Hail Mary — With CTO James Reggio (acquired for $5B by Capital One!)

We have 3 pillars for our AI strategy. We have our corporate AI strategy, which is: How are we going to adopt and buy AI tooling across the business and, basically, every single function to 10x our workflows? Then we have our operational AI strategy, which is how we're going to buy and build solutions that enable us to lower our cost of operations as a financial institution.

The final pillar is the product AI pillar: Are we going to introduce new features that enable Brex to be a part of the corporate AI pillar of our customers? We want to build features and be a solution that somebody else is saying to their board, “Hey, we adopted Brex, and this is part of our corporate AI strategy.”

Hey everyone, welcome to the Latent Space podcast. This is Alessio, founder of Convex Labs, and I'm joined by Swyx, editor of Latent Space.

Hey, hey, hey. And we're here with James Reggio at Brex. Welcome.

Hey, thank you for having me.

Thanks for visiting from up in Seattle. I've been up there a little bit. It's cold up there, huh?

Yeah, and we have an atmospheric river hitting the city right now. We're getting the full-on winter effect right now.

Well, you're here. We're talking about the AI transformation within Brex. There's a lot of interesting tidbits that we're going to draw from your article, but also your background. You've got a wide array of experience, from Stripe to Banter to Convoy, and I'm mostly interested in your journey as one of the rare people who've transitioned from a mobile engineering leader to a CTO, which I think is also a bit more rare.

I used to have this comment in the past where there's a career ceiling for people who work on client-only things, where usually they don't hit CTO, whereas they typically promote the backend people—the backend, cloud-focused people—to CTO.

Yeah. It's something that I hear fairly frequently, because there aren't that many folks with a front-end background who reach this level of leadership, and it's exciting for me to be able to represent that group. I'll say that even though my résumé kind of reflects that I've been more on the front end of things, it's probably more my experience as a founder a couple times over that actually helped me get to this level of my career working for somebody else.

Becoming CTO was very much a leadership and general business role as much as it is a technical role. I think it was more the skills that I built from starting companies and trying to build those up that made me a decent fit and enabled me to get the nod from Pedro to take this on, as my predecessor left about 2 years ago.

One thing I'm curious about—this is a little broad and unscheduled—but a lot of startups are bragging about how many ex-founders they have. To some extent, you want people with the founder mentality and agency, which is what you want your employees to have, to take initiative in the company. But I also wonder if it's becoming an anti-signal sometimes. I don't know if you've thought about this.

I think it's more about the churn for me, especially when people are hiring ex-founders. If you're truly of the founder gene, it's kind of hard to just stay somewhere as an IC for too long.

And then it's like, all right, I joined this thing, and then in 1 year I'm back to being a founder. I'm curious for you: What was your—I'm sure you thought about leaving and doing another company instead?

In fact, that was the alternative. Even at the time that I got the phone call where they made me the offer to become CTO, I was thinking about leaving to go start a company. What's interesting about it is that we actually launched a new recruiting and employee value proposition for Brex a couple of months ago called “Quitters Welcome,” where we intentionally lean into this idea that we have a disproportionate number of folks who go on to become founders or heads of a department when they leave our company, and we celebrate that.

It's something that I'm very proud of, and that means that we welcome in people who want to get a different experience. I think there are certainly a lot of founders who don't make it—who don't scale their own businesses to the scale that we've achieved at Brex. So there's something to be learned when they come in. Then we're very happy to support people on their way out.

The value proposition I find most relevant, because a lot of the folks we're hiring as AI engineers are either winding down their companies or considering maybe running an AI startup, is that we can often give them interesting problems to solve—problems that maybe they even want to build their own startup around—but with instant distribution.

That is the allure: You can come into this business and build financial AI applications and instantly have that deployed to roughly 40,000 customers, from the Fortune 100 down to tens of thousands of startups. That's what I think is appealing to the founders. But the challenge then is making sure that we set them up for success in an environment that still feels a little bit like the startup that they might build themselves, versus something that's too corporate.

Yeah. Instead of doing your own company and then coming to Brex and being like, “Can I integrate into Brex? Get all the data?” How's the engineering team structured?

We have about 300 people in engineering, around 350 total across EPD. For the most part, we structure around our product domains. Brex is a corporate card; it's also a corporate bank account, expense management, travel, and accounting.

We have full-stack product domains that are roughly 30–40 people each, with everything from low-level infrastructure up to the web and mobile experiences. That's generally the structure of our engineering organization.

Naturally, we have an organization that focuses on infrastructure and security. Then there are 2 additional centers of excellence that we've built that kind of violate that org design, where we've felt the need to put more focus or operate slightly differently. AI is one of those areas where we have another team of roughly 10 people focused primarily on LLM applications.

This is actually something we did this summer: We paused and asked ourselves, on our AI journey toward infusing our product with AI and generating customer value, what would a company founded today to disrupt Brex look like? We then tried to use the answer to that question to form this team internally.

It's a little bit off to the side. Ideally, everybody comes up to speed and contributes LLM features, but we have this off to the side right now in a centralized manner.

What's the difference in AI adoption for those teams? Are the people on the LLM team much bigger Cursor users, Claude users, or do you see similar diffusion?

It's actually fairly uniform across the entire engineering department. It's kind of funny: one of our largest Cursor users is actually an engineering manager. I think this also speaks to our core value of “Operate at All Levels,” where we want all of our EMs and everybody in leadership to still basically do the job they manage.

The journey of getting everybody into using agentic coding was not exclusive to the AI group.

Yeah. In fact, I think this podcast was actually set up because I cold-outreached Pedro because he tweeted this—I assume this is in Brex. He says, “I started a new company inside Brex to build the future of agentic finance. No BS, just builders building 9-to-6 and pushing production-grade agents to 30,000 finance teams”—now 40,000.

Then he actually has a little job description, which I think is really interesting. I'll skip that and go straight to: “Brex accelerated to grow 5x and cut burn 99% in the past 18 months.” I assume that's a mix of internal AI automation and other stuff. Basically, I wanted to put some headline numbers up front to impress people before we dig into the details.

Yeah, absolutely. And you're correct. That's the team that we have, this AI team.

Very young team.

Yeah, it's very young. The composition of the team has been really interesting: It's very young, AI-native 20-year-olds paired with more staff-level software engineers who've been here a while and can navigate the existing codebases and understand the product and customer deeply.

We've formed these really tight-knit pods in the AI org. They're generally 3 people: somebody with more of a product- or customer-focused background, a staff engineer who knows where the skeletons are, and a much younger, AI-native engineer who can do things with agents that the rest of us dinosaurs maybe either don't dream of or can't.

I think part of it is that sometimes too much experience or too much knowledge of how to solve a problem can actually be an impediment to thinking differently about it and thinking about it from an AI-first lens.

But yes, we’ve been slowly growing that team in the same way that, like a pre-seed startup, you want to be very careful about talent density and be very deliberate—only hire when you absolutely need it. At this point, it’s just about 10 people. I think it was probably 4 or 5 people when everybody was actually in the photo that was attached to that tweet when Pedro put that out a couple of months ago.

Yeah, we’ll put it up. It’s a photo at 1:20 a.m. on a Friday.

Yes. Oh, yeah, because we always do Friday demos, and that’s a time for everybody to get kind of executive review time.

Everyone’s in Seattle.

Those folks were all in Seattle, but they’re actually geographically distributed. We have a couple of folks here, a couple in São Paulo, and a couple in Seattle.

How do you address this? We have this AI center of excellence, which is basically the people running these teams across the company. How do you make the other engineers not feel like they’re not special? I think that’s something that I hear a lot: “Why aren’t these people working on all the cool LLM things, and I’m stuck working on the KYC integration with whatever?” You know what I mean? How do you build that culture?

You know, it’s interesting. I thought that would be more of a problem, but the benefit of having really optimized our engineering culture around business impact actually causes it to cut in the other direction. Some folks don’t want to work on the AI products because they don’t have as much clear, direct business impact right now. It doesn’t impact revenue as directly.

I think, for the most part, we’ve enabled folks who have a strong desire to work on AI products to join that team. Somebody transferred out of our expense management organization to come over there because they were really passionate about taking their knowledge of policy evaluation and bringing it into the AI team.

For the most part, I think everybody understands how their work ladders up. Maybe there’s some friendly rivalry, because the folks who work on, say, a card product drive 60% of our direct revenue, so they’re pretty happy with that and they don’t feel like they’re being left out.

I will also say, as you probably saw in this piece that we put out with First Round, there are a lot of smaller applications of LLMs peppered throughout all of our product and operations teams. It’s just some of the more novel, agentic layer that sits on top of Brex that’s been put together in this sort of isolated team. It’s not like folks aren’t getting to build with LLMs or use LLMs on a daily basis.

Maybe run people through the Brex agent platform. We’ll put the diagram in the video, where you had the LLM gateway and the whole MCP layer. We just had the creator of MCP right before you, so this is very timely. How did you start building that? What’s the architecture?

Yeah, the architecture—you know, I think simple is elegant. We’ve had basically an LLM gateway and a basic control plane from the very early days. In fact, right before being tapped to become CTO, I was leading an AI labs team internally. In the wake of the announcement of ChatGPT, everybody saw this new technology and said, “Hey, what are we going to do with it?”

One of the first things that we did—I think January 2023 that would have been—was try to put together some internal infrastructure that made it possible for us to deploy, manage, version, and evaluate prompts, and then manage data egress and model routing, and have some very basic observability and cost monitoring in an LLM gateway.

That’s infrastructure that we stood up, and it still continues to power a lot of those smaller, more precise applications of LLMs. For instance, we’ve set up a completely automated pipeline for evaluating customer applications to get them onboarded instantly to Brex, which used to require human intervention either for underwriting or KYC. Now we basically have a series of agents, and particularly research agents, that will go and do the work that humans would normally do. That’s running on top of this hand-rolled framework.

For the agents on Brex that we announced in our fall release—which is this agentic layer that we’re building that sits on top of Brex and can embody workflows that a finance team would normally hire humans for—we’ve actually started using Mastra as the primary framework for accelerating us. They’ve built everything in TypeScript, which is another technology choice that answers the question of what we would do if we started Brex today, but that isn’t the case for all of our existing backend code, which is either Kotlin or Elixir.

We have a mix of pgvector and Pinecone, and I think what we’ve seen is that we’re always reevaluating the tech and framework choices as we go, because the half-life of code has declined so significantly with agent coding. It’s actually quite easy for us, and for anyone else, to try on for size a variety of different pieces of tech to figure out what’s going to be most ergonomic for solving the problem.

Double-click on Mastra. That’s a new choice, an interesting one.

Yeah, I think the main reason that we adopted Mastra is that it provided the ergonomics that we were actually looking for. The ergonomics of Mastra are quite similar to the internal LLM framework that we built 2.5 years ago, whereas LangChain was available at the time—2.5 or 3 years ago—but it didn’t quite feel right to us when we were trying to use it. It kind of addressed things that weren’t the pieces we needed to address, which was being able to have really simple observability, logging, and tracing.

LangChain didn’t do it.

I mean, at that time it didn’t. I think it was really—

Well, they fixed that.

Yeah, no, they certainly did. But we evaluated LangChain, turned off of it, built our own thing, and then, as we were looking, we wanted to deprecate this internal framework that we built because it wasn’t leveraged enough for us to maintain it. Mastra ended up fitting the bill for the feature set that we were looking for.

What’s been interesting is that about half of the applications that we’re building right now on the agent layer are running on Mastra, and then the other half are actually still running on yet another internally developed framework. That framework is focused more on networks of agents—sort of multi-agent orchestration—versus more strict, single-turn workflows, which are easier to use with either LangGraph or Mastra.

Tell us about your multi-agent framework. What are the design considerations? Why is this the first we’re hearing about it?

Yeah. It’s funny: a big reason why we haven’t written more about this is that it continues to evolve quite a bit. I feel like we actually had a blog post that we were going to put out in conjunction with the fall release, talking about how we built this, and by the time that we finished the blog post and had all the packages ready, it was already halfway outdated.

The way that this has started to emerge—this multi-agent network approach to implementation—was when we were trying to scale up our sort of consumer-grade Brex assistant. If you think about Brex and our customers, there are really 2 very broad personas that we serve. We serve members of a finance team, who are generally going to be in roles like accountant, controller, or head of FP&A. For those folks, they’re going to be interacting with agents that are much more specific to their roles.

But then the other broad cohort of users we have are employees of companies that have deployed Brex. You join a new company that uses Brex, and you get your Brex card. Our goal for employees is for Brex to completely disappear. The best UI/UX for Brex is just the card. Every single thing that you have to do in the software beyond just swiping the card is an opportunity for AI to eliminate some work for you.

What we thought was the right approach to solving that was to embody an executive assistant for every employee. As an executive at Brex, I have an EA, and she knows enough about me. She has access to my calendar and my email, and has all the context on when I’m traveling and for what business purposes. She’s basically able to do everything that I would be obligated to do in Brex, be it booking travel or doing expense documentation.

We wanted to build that EA connected to the same data sources and see if we couldn’t simulate that behavior, so that basically your interface to Brex is SMS and the card. When we started building that out, the most naive architecture for that would be to have an agent with a variety of tools and maybe do some RAG to ensure that it has appropriate context for the conversation.

But what we were finding was that the wide range of different product lines that exist at Brex made it difficult for 1 agent to perform well while being responsible for everything from expense management to finding and booking travel to answering policy and procurement questions. That’s when we started breaking down the problem into a variety of subagents that sit behind an orchestrator.

Obviously, this isn’t something that can be implemented using LangGraph or Mastra, even though they have the notion of these as network switches in beta. When it came to building evals for the system, we hit the eject button and built our own framework. In this framework, agents are able to DM with other agents and have multi-turn conversations among themselves to coordinate and complete a task or objective.

What’s been nice about that is that you can have your Brex Assistant as 1 single point of contact between you as an employee and the Brex product. Behind your assistant, if the company has expense management turned on, you have an agent for that. If they have reimbursements, there’s another agent for that. If they have travel attached, there’s a travel agent for that.

It also facilitates our conception here that software encapsulation patterns generally get projected into the agent space. It makes it easier for us to have the team that owns and understands travel be the ones to iterate on that without needing to redesign the total system or needing 1 team to own every possible action you could take as an employee.

I’ll say that I’m still of the mindset that somebody will build a great framework and we’ll ultimately migrate to it. Or it might be us—we might ultimately open-source this. But for us, this has worked out quite well in lieu of a couple of other approaches that we tried along the way that just didn’t perform well.

One approach was to overload the agent with a variety of tools. Another was context switching, where we’d say, “This conversation looks like it’s more about reimbursement, so let’s update the prompt with more reimbursement context.” That approach didn’t perform as well as actually having a reimbursement agent that could collaborate with the assistant.

What about MCPs as subagents? Oh, yeah, that’s another pattern.

The key thing there is that there’s actually a lot of value in having multi-turn conversations from the orchestrator, or the assistant, to the subagent, whereas a tool call is basically just 1 RPC. Oftentimes, what will happen is that the user reaches out to their Brex Assistant and says, “Hey, how much am I allowed to expense per person for dinner tonight? I’m taking my team out.”

The assistant is then going to reach out to the policy agent. Maybe the policy agent needs to know whether this was a customer event, a team event, or whether the user is traveling. It may not be able to answer the question immediately, so it’ll reply to the assistant and say, “Hey, I need you to ask this clarifying question.”

The assistant will then return to the user, ask the clarifying question, and they’ll have this multi-turn conversation across multiple agents instead of it being encapsulated in a single call-and-response tool call. All the subagents still have a ton of tools, but I think of MCP and tool usage as the interface to all of our conventional imperative systems, not the AI space.

Yeah, that’s the conversation we were having earlier: whether or not it should be an agent-to-agent protocol as well.

Yeah, there should be a chat back.

Exactly. Exactly. And that’s the thing. One of the ways that we actually grafted this into Mastra before we built our own framework was to make every subagent a tool. The input was just natural language, and the output was natural language.

If you needed to have multi-turn conversations, you would basically just put the full conversation in as you kept calling the subagent as a tool. At that point, you’re like, “Okay, the ergonomics are kind of—the framework is fighting me on this.”

It’s actually helpful for us to conceive of it as an org chart. It’s an agent org chart, with my EA DMing other specialists and having brief conversations to support me as their client.

Yep.

That was a really good deep dive. Thanks for indulging me. I feel like you guys aren’t afraid to make your own tech, which I think is a competitive advantage. I really like that culture.

Maybe we should go a bit breadth-first as well. Of course, I think we also deep-dived a little too much into 1 area. We’ll put up the chart, but I’m also very interested in the internal agent stuff, the operational stuff, and just the general platform scope. So please feel free to go into your spiel on it.

Yeah, of course. One of the things I was trying to do at the beginning of the year, as CTO, was articulate what our AI strategy was as a business. Every member of our board was asking, “Hey, what’s your AI strategy?” And while we were doing a lot of this, we’d literally go, “He’s got it.”

Well, yeah. [laughter]

Yeah. And if I didn’t, I’d be in trouble. I think he was also counting on me, given that I was doing the AI organization before becoming CTO, to have it.

That’s true.

But a big part of it was that we were doing a lot with LLMs. It was more like these little 1-off features: maybe mixing in some suggestions here or doing a little bit of ops automation over there. But it wasn’t easy to create a verbal framework for all of these investments. Without that framework, we weren’t able to set a vision or a roadmap for our investments.

What we did at the beginning of the year was take everything that was going on, as well as all of our ambitions, all of the good ideas, and all of the problems we were trying to tackle as a business that year, throw it all on the table, and see if there were ways to cluster it into a framework that made sense to the business, to our board, and to ourselves.

We came up with what I think is not particularly novel, but has helped us quite a bit. We have 3 pillars for our AI strategy.

We have our corporate AI strategy, which is how we’re going to adopt and buy AI tooling across the business, in basically every function, to 10x our workflows. Then we have our operational AI strategy, which is how we’re going to buy and build solutions that enable us to lower our cost of operations as a financial institution.

I think it’s fairly intuitive: financial institutions like ours face a lot of regulatory expectations, and there’s a high ops burden for running our business. That includes internal use cases like fraud detection, underwriting, KYC, and dispute automation on card transactions. Those types of operational investments make up our ops AI pillar.

The final pillar is the product AI pillar. Are we going to introduce new features that enable Brex to be a part of the corporate AI pillar of our customers? We want to build features and be a solution that somebody else can point to when they’re saying to their board, “Hey, we adopted Brex, and this is a part of our corporate AI strategy.”

It has this nice little feedback loop. Within the company, we basically split up and did a little bit of divide-and-conquer. Folks in IT and on our people team were spending more of their effort driving corporate AI: making procurement decisions and creating a culture of experimentation where we spotlight and incentivize people for trying to improve their personal workflows using AI.

The pieces that I’ve been more involved in have been operational and product. We were just talking about product, which is the agents on Brex and so forth. But I think the operational AI investments have been some of the most immediately impactful to the business.

We have hundreds of people who work in our operations organization, and it’s something that differentiates us because our CSAT and the quality of our support and service are very high. It’s something we’re very proud of. We’re trying to figure out how to automate a significant portion of this and use LLMs in a way that doesn’t degrade the customer experience, while also addressing the future roles of the people who already work full-time for us.

This is where Camila, our COO, who co-wrote the piece with First Round with me, has been leaning in really aggressively to help every member of the operations organization start rethinking their role. They’re not people who simply execute against an SOP; they’re people who are going to build prompts, build evals, become more AI-native, and change the way they do their work.

And so a lot of the engineering we've done has been to enable folks, say, in fraud and risk to refine prompts and add additional automation to their workflows.

Yeah. And this secret fourth pillar—the platform.

Yeah, yeah, exactly. That is the thing that ties it all together. Exactly. It is the platform.

I think what's been really nice is that even though the platform is kind of a loose term because it consists of a wide variety of technologies, as I said, we haven't been too religious or dogmatic about everybody needing to be on one particular thing. What we've seen is that by making a variety of ergonomic options for building with LLMs available, it has really made it easier for us to make a quick leap forward on operational AI.

As soon as we put our mind to it and said, “Look, we want to hit an 80% automated acceptance rate for all startup and commercial businesses that apply for Brex. We want a decision within 60 seconds that's fully touchless, with no humans involved,” we were able to break that down and actually build the agents and the tools on top of that platform really quickly. A lot of those tools are the same tools that our product AI agents use as well.

I was pretty sold on ConductorOne. I don't know if this is under exactly that bucket.

The provisioning command—I was like, “Yep, I want that.” Yeah, that was actually—I’d love to talk about that. That's actually on the corporate side.

I think this goes back to maybe another intuitive but bold decision that we made, which is that we're not going to try to pick winners in the horse race between the foundational model providers, the agent coding tools, or basically anywhere where there's an active horse race.

What we do instead of trying to pick a single solution is procure a small number of seats for multiple solutions and give employees the ability to pick whatever one they want to use. For instance, we allow employees to go in Slack and use ConductorOne to get a ChatGPT, Claude, or Gemini license. You can basically build your own stack where you pick your chat provider.

As a developer, you can pick between Cursor, Windsurf, or Claude Code credits, and you can basically craft your stack to your preference and easily switch between them. What that does for us, too, is that although we have enterprise agreements in place for all of them—for the privacy and non-training guarantees—when we go to renew these contracts, we can resist the need to do a wall-to-wall deployment.

We can say, “Hey, look at the usage trends. Our employees are voting with their feet. They're voting with their dollars. Maybe your tool isn't as hot as it was a year ago.”

Does it give you a dashboard of what people are choosing?

Yeah, actually, we look at that. We were looking at that as we're going into budgeting for next year. Very interesting.

I would love to see what's really up and what's really down.

It's fascinating how different the landscape is every 3 months. I think one of the interesting challenges we had early on was getting folks to just try these tools and try to incorporate coding.

Early on—12 to 18 months ago now—we had to get folks to take the time to try a new workflow. At this point, I think what we're seeing is that even when a new model hits the scene, like when Codex came out and everybody was saying, “Codex is better at code generation, but it's a little bit slower,” fewer folks are kicking the tires on new things.

They're just so comfortable with the ergonomics of their current workflow. Some folks are saying, “I just want to stick with Claude Code because I know it now. I've been working with it for 9 months, so I don't need to keep switching.” I don't feel the incessant need to keep trying new things because I've gotten comfortable. I'm an iPhone person, and I'm just going to stay with an iPhone, even though there's some really sexy Android hardware out there.

Do you have one of the big numbers, like 80% of all our code is written by AI, or how do you measure it internally?

Yeah, no, not really. What we do is measure the attributions on the number of commits that have the “Co-authored-by” attribution, and we pull some of those stats. But I don't index heavily on those at all.

I don't know how I'd honestly calculate that number. I agree.

Yeah. The thing that we're really focused on now is that we're at the point in our AI agentic coding journey where we're trying to solve the second-order effects of a little bit too much slop—maybe a little not enough rigor in code reviews.

The adoption is there, and now we have to figure out how to mature our usage of these tools so that quality and long-term maintainability don't suffer. One of the other facets of being able to generate a lot more code more quickly is that the drift between team members, as far as understanding of the code in their services, increases. Everybody's moving faster and more independently.

That's another risk that we're starting to see in incident response, where folks don't know a service as well as they used to because it's changed so much in the past couple of months. Everybody's moving more quickly.

Yeah, this has been a major topic for me this year: codebase understanding and slop. Obviously, it's so much easier to generate code, but now we have to review it. To some extent, you can't really fight AI with more AI. You can't just throw an AI reviewer on AI-generated code and say you've solved it.

You do need to scale human attention. I think that's something I've been pushing a little bit in terms of every engineer owning more code, period, and being parachuted in and expected to ramp up, be productive, and also fix bugs. If you're on pager duty or whatever, you have to do that because everyone's going to try to be more efficient, and you're supposed to see ROI in productivity. If you don't, then what's the whole point of this?

Exactly. Exactly. I think it's funny, going back to the point that you could add AI on top to solve the problems that AI introduces, but you just keep going. That's an endless chain. And so that's a little bit of the tension there.

I've been thinking a lot about how the craft of engineering is evolving, and I will say that I feel further away from being able to predict what it looks like than I did this past summer, when I spent a bunch of time building alongside our AI team.

I actually went on leave for a month and joined the AI team that we were building just to build alongside them. I felt like it was really important for me to deeply understand the problems in the tech. I was writing and pushing code, effectively 996, and I went through so many different moments of realization: “Oh my God, this is going to change everything,” to “Oh my God, this is just amplifying all the good and the bad in the industry,” to “Oh my God, engineers aren't going to have a job anymore.”

I felt like I had all the predictions back then. At this point, I'm just very interested to watch the phenomenon continue to unfold in front of us. I was chatting with a bunch of really bright college juniors and seniors at a dinner we hosted last night. These folks are about to enter the industry, having come up in the era of agentic development and LLMs.

I asked them, “What is your workflow when you're building a project? How do you use agents, versus when do you decide you're going to actually just write code by hand?” I was surprised to hear that the consensus was that most people there were using agents to collaborate on building a design document and on the architecture of the solution they wanted to build.

Then they might ask the agent to emit a document or an implementation plan, but they would go and write a lot of the code themselves. It was more of the rubber-duck, co-architect use case that was most prevalent in that group. I was very surprised by that.

I'm impressed. The kids are all right. Yeah, I know. They still want to actually write the code themselves. It's interesting.

Yeah. What we hear from the Gen Zs at OpenAI is that they just YOLO everything into code.

Yeah, I would say most of the code I generate is...

But I spend a lot of time on the doc. It's curious: when you're younger in your career, you don't really have all the mental models of the different patterns to instruct. So I feel like there's an overreliance, especially if you're doing the design doc. I feel like most senior engineers will spend more time on that. Even things like, what columns should you index?

It depends on what queries we usually run on this table and things like that. It's hard for any AI to know that, right?

I feel like the role of the more senior engineer should actually be more of this. It's spending time teaching the AI, and then the AI can teach the junior people in a way.

Yeah. Yeah. Everything looks like mentorship and management. [laughter]

At the end of the day, right? It's like you're breaking down tasks, you're supervising work, you're giving feedback. It's basically management.

Except that agents are really bad at memory still. They basically have zero memory, and it's Seattle, 2025. What's going on?

Yeah. [laughter]

Yeah. What's your internal stack for preferences? There's explicit preference you can use with AGENTS.md and all that stuff. There's implicit preference with lint rules and things like that, where it just happens—you don't have to tell it. How do you structure that?

Oh, are you talking about agentic coding or memory as a platform?

Yeah. Yeah. For coding specifically, and then we can talk about the whole Brex platform.

Yeah. Just nothing special—just a lot of explicit rules.

MD files.

Yeah. And then we have linting. We still have traditional linters in place for a couple of different language toolchains. And we're big fans of Greptile, and we use them for basically all of the smarter-than-linting code review. That's been the 1 solution we've aligned around that has served us extremely well.

Yeah. Greptile.

Yeah. No, we're huge fans. They've built something really impressive. And I think the thing that constantly blows my mind about it is the way that they're able to have a really impressive signal-to-noise ratio. The comments that it leaves are very, very high-signal. I never regret going through all 65 comments it leaves on my diffs because it catches so many things.

Yeah, I found the Codex review to be really good. I don't use Codex for code generation, but the review product is very good for some reason. When I was working in Rails, there was this project called Danger. It was kind of like a semantic linter.

Exactly.

I feel like there should be more of that now. The rules are one thing in generation, but I want something in my CI that enforces these rules and calls out where they're broken, and then I can just copy-paste that into an agent.

Yeah, when we started building this new agent codebase, we were answering the question, “What would you do if you built a Brex disruptor today?” It wouldn't be to pick Kotlin and Elixir as the backend. So we actually went with the full TypeScript stack, and we were building on all public interfaces and really trying to make sure that this agent layer was at arm's length from the good and the bad of the core of our product.

One thing I think we did early on—and I don't actually know if this is true because, again, the team keeps iterating—is that we're having good luck using Claude Code in a GitHub Action to basically do more of that Danger-style code review. So we have a prompt for it that goes through all of the different facets that are more conceptual versus rigidly enforceable by a linter, and has it leave a big comment at the end with your conformance to the idiomatic coding patterns of the new repo.

I wanted to spend some time—you said you wanted to dive into operational agents: customer support, onboarding, KYC, fraud, delinquent account disputes. This is, I imagine, the bulk of the work.

Anywhere where there's a good story about when you started out, it was going to be this way, and then you discovered through building or through customer contacts that it had to go a different direction. That difference in beliefs is something that people can learn from.

The thing that immediately comes to mind is that we believed at the beginning that using RL for credit decisions—credit and underwriting, like how much of a limit should we give to this business—would be the way that we would end up. Reinforcement learning would be the way that we would go about building a model that effectively made decisions in the way that a human underwriter would.

And it turns out that it wasn't. We made this big investment. We were working with an outside company that specializes in this, and the performance we ended up getting was inferior to just building a web research agent.

So I think what we took away, what has been most evident in operational AI, is that in operations you need to be able to break down problems really granularly and be able to form SOPs that humans can repeatably follow and thus can be audited. So much of the responsibility in operations is to have auditable, repeatable processes that help to ensure that we're operating in a compliant manner, and that actually translates so cleanly to LLMs that we haven't needed to use too many sophisticated techniques in operational AI.

It's been relatively simple. New tools, agents, or maybe even a lot of problems can be solved with just a single-turn chat completion. We did 1 attempt to overengineer and use more sophisticated techniques, and we discovered that, in fact, the solutions are a bit more plain and less technically sophisticated.

The challenge is really articulating and refining prompts to reflect the execution of the SOP and all the institutional knowledge that isn't written down, so that agents can properly replace the humans or contractors we would have making these decisions.

How do you decide what is worth spending a lot of time building versus what you think some of these models are just going to be good at? Some of these tasks are so generic. They're not really about Brex, so you can assume the models will be good at them, versus some of them that are very specific to you.

We prioritize the tasks that are most common for the broadest number of customers. Some of them are fairly intuitive, like being able to research a customer to assess the legitimacy of the business and whether that business would fit our ideal customer profile for onboarding, because there are certain types of businesses that we either legally cannot serve or are not comfortable serving. So that's the type of really basic research and a relatively straightforward problem that isn't hyper-specific.

The things that are a little bit more specific to us or to companies in our sector would be preparing documentation for a card network dispute. If you go and dispute a transaction on your personal card, you will provide evidence to your card issuer. The card issuer then has to put together a 3- or 4-page Word document that goes to the card network and then eventually goes to the acquiring bank, and all of that is much more specific to our business.

It's a huge operational overhead for us, and that's something that we decided to automate later because it's not on the critical path of serving the vast number of our customers. Disputes are expensive but not very common operational processes, so they're lower on the stack. I think we're getting there right now, but this year has basically been us looking at every single process and stack-ranking them.

The thing that got us started down this path was that we wanted to expand our ideal customer profile to support more businesses—a wider variety of commercial businesses—which tend to be businesses that aren't growing as quickly. They're not tech startups, which have a lot of growth, and they're not usually enterprises, which also tend to have a lot of growth. It's more like a law firm or a dentist's office.

These are the types of solid businesses that we should be able to serve and underwrite, but the cost to onboard them and the cost to serve them if you have all the humans in the loop make them ROI-negative. So that was the first use case of AI within our ops organization, which then led to us really understanding that we could automate much more than that.

Is this Brex going back into SMBs?

That's a good question. Yeah. So never let that die.

I think the way we've thought about this is we want to always offer our product to customers where we believe we have an offering that is well-suited to the needs of those businesses. And I would say that, still, for very small businesses, our offering isn't built for that.

It's built for companies that have some degree of scale, typically with at least 1 person, if not a couple of people, on their finance team. We consider these to be more like the commercial segment. It rhymes with SMB, but our approach back then was a little bit more naive. I would say we were also just going for a volume game there.

Our internal controls were not as strong, and we didn't have as much experience underwriting those businesses. It really ended up being a huge burden for the business, almost existential for us, to have those tens of thousands of customers that were all ROI-negative. We're trying to scale to serve more businesses outside of tech and outside of the SMB market segment, but do it thoughtfully.

I think right now our minimum threshold is $1 million a year in recurring annual revenue, or $10,000 or more per month in card transactions. That's kind of the low end of our ICP, which is obviously not what you would think when you think of a small business. Small businesses tend to still be smaller than that.

Oh, wow. That's really small. Okay.

Yeah.

Yeah. Mid-market.

Yeah, exactly. And it's funny. It's just the names of these segments. I don't know.

No, I think I like that. That's lower mid-market. And it's funny, though, because what we call enterprise—what sales calls enterprise—might be a business that Salesforce calls mid-market, right? It just depends on the scale of your business when you use these terms.

And all of these things are built in the Brex agent platform, like all these automations that people build.

Yes, exactly. In fact, most of the operational AI is running on that original platform that we have. One element of it that I didn't mention is that most of the UI and UX for this platform is built in Retool. You can basically go into Retool, and there's a prompt manager, a tool manager, and an email manager. That's where much of this was built.

The goal was, again, to make it more accessible and more ergonomic to get started. A secondary effect of having a more visual set of tools is that it's enabled members of the Ops organization to do prompt refinement themselves. You don't need engineers to refine the prompts or even test new foundation models when they come out.

I think that's another fun thing: when a new model drops, folks will go into the platform and run the evals on the new model to see whether they can get better performance, or whether it has different latency, cost, or other characteristics.

Yeah, you want the domain experts, or the people directly using the tool, not the engineers who are somewhat removed from the tool. I do want to highlight to listeners that a lot of the Brex agent platform is just things that every company should have, basically: a prompt management system, which we talked about, where the domain experts are doing it; multimodel testing; evaluation and benchmarking frameworks; API integrations for automated workflows; and an MCP-based architecture shared with Brex's external AI products. This one is obviously very Brex-specific: a knowledge base for understanding Brex's business.

Yeah. Do you want to expand on that?

Yeah. This is an area where we've only scratched the surface, but a big challenge that we face is that the world knowledge, or the knowledge that's built into the model about what GPT-5 thinks Brex does and how it thinks our business operates, is quite different from what our business offers today or how our product works.

We've had to work on building a corpus of product documentation and process documentation, and curate this set of information to ground a variety of our LLM applications, including the Brex Assistant, which is the assistant that employees will talk to. We don't want it to hallucinate features that we don't have or give wrong information there.

Similarly, some of the operational agents need to be grounded in what our ICP is. If you ask GPT-5 right now what types of businesses Brex onboards or serves, it might not give an accurate explanation. It might say, “Hey, we're a corporate card for startups,” which is what we did 7 years ago. It might say we only serve enterprises.

That's been an interesting challenge. What we've been trying to do there—and I'm actually going to be spending time with folks talking about this internally next week—is figure out whether we can refresh our strategy and unify it. We have a lot of product documentation that's internal for our operations and go-to-market teams. We have a bunch of product documentation that's external for our customers. We have a lot of go-to-market enablement material that's more sales-pitchy, and we have documentation that's put into Sierra, the chat assistant that we use for frontline support.

Ideally, all of this could draw from the same source, but right now it's a little bit fragmented. It's something we're trying to invest in, though, because at the end of the day, the duplication of efforts is wasteful, and it's absolutely necessary to get this right.

Just to disambiguate: Sierra, meaning the Bret Taylor startup?

Yes, exactly.

Yeah. I would expect that, since you've built so many other agents, that's one you could build yourself.

That's like solving problems that are not differentiated enough for us. What's interesting about Sierra, and what has been really helpful, is that the UI and UX of administering a Sierra agent is really accessible for the Ops and CX Strategy teams. It's much more low-code and more workflow- and DAG-oriented, and we have engineers giving it tools to take actions.

For the most part, it's nice not to have to build the UX for somebody to manage something like that. I think the fact that Sierra speaks the language of customers—yeah, exactly, speaks the language of CX—and can do all the reporting and telemetry that our VP of CX would like to see, it's just one fewer thing that we have to build.

What about evals? How do you build evals, and who manages them?

It depends on the application. On the operational AI side, those evals are basically baked into the platform around every prompt or every agent. For the most part, these use cases come online like this: the V1 of our commercial underwriting agent, or the V1 of our startup KYC agent, is co-developed between a subject matter expert in KYC Ops and an engineer. They're going to co-develop an initial eval set.

From there, generally in Ops, you're always doing QA, whether on humans or on LLM decisions. Whenever there's a mistake, that's almost always going to result in another eval being written as a regression test. All of that within Ops AI is pretty straightforwardly managed.

On the product AI side, that's where it starts getting more challenging, because the multi-agent network is quite challenging to evaluate. We try to adopt some of the state of the art for multi-turn evals. We'll basically have an agent embody the user, and the end-user agent is given an objective. Then we have it run a multi-turn conversation and use an LLM as a judge at the end to do all of the different assessments.

The other technique we use that's interesting is that sometimes you don't want to do a full multi-turn eval. Those are kind of like integration tests; they sometimes test more than you want to assess. Sometimes we'll pre-seed an initial preamble to a conversation, or maybe a couple of turns will be handwritten. We'll set the eval to start there and see whether we're able to isolate certain behaviors.

It's still a work in progress. At the end of the day, a lot of it is just periodic human review and looking at cases as we go to summarize them. We'll reflect on a conversation after a certain amount of time has passed, summarize it, and extract facets such as whether it seemed like the user accomplished their objective. Then we'll manually review a lot of the cases where that failed and decide to write another eval for it.

Are all the evals supposed to pass, or do you have a set of evals where someday the model will be good enough? How could that change over time?

Yeah, it's interesting.

I don't know if we have any that are like, “Someday, I hope it'll be good enough to do this,” but there are evals that are blocking because they would indicate a regression—an unacceptable regression. These tend to be accuracy-related evals, but then there are others that are more about tone and coherency, and these types of things, where we're just looking at those over time as a metric.

The team is actually interesting. I think we're going to get a big update on how the team is thinking about eval at our Friday review. This is an area where I'd say the largest challenge—the largest change we needed to make in how we were executing, sort of as a lab or an incubator, back earlier this year to where we are now, where we've shipped and we're trying to increase the rigor—has been around avoiding regressions and having increasingly robust evals.

Yeah, I work with a company called Vijil that does user simulations, and I think that's what's been interesting. Some of these things they just don't expect—the customer does not expect the model to do—

But they want to track the sophistication of the model in a way, if that makes sense.

I feel like most companies know what they don't want to happen, but it's almost like they cannot quite articulate, “Oh, I want, in the future, the model to be able to do this.” They can do it today, but I'll keep running this eval.

That's actually really interesting to me. I'm going to take that away and start thinking about this because there are going to be certain—I mean, we've already seen this—where users will ask the assistant for help with things that we don't support yet or haven't implemented yet. Those are opportunities for us to effectively write a test that's going to be failing for weeks or months and eventually will go green, but is a way for us to show the progression of sophistication of the assistant. I really like that as an idea.

I wonder how you also catch hallucinations of things that it doesn't have. That's usually the problem: it'll pretend like it can assist with something. One thing that is really annoying and has been tough to prevent is that the assistant, because it's used to speaking to other agents that can support it in accomplishing various tasks, will hallucinate if you ask it to help with a task that it thinks it probably should have an agent to work with.

It'll say, “Oh, yes, I'll reach out to the finance team on your behalf to pass this question along,” but it's not doing anything. There's no finance team. There's no way for it to do that. This comes up a lot: “Would you like me to ask the finance team?” There's no actual tool guards for that.

Yeah, yeah. That was something that we had to react to. Oh, no, but we don't. I think we've been able to beat that with a system prompt. We don't have as many guardrails in place right now around a couple of potential things that could get us into trouble.

Really? It's surprising. When I was first kicking around the idea of all these things 2 years ago, I would have said that guardrails would be more prevalent, especially in finance use cases, but surprisingly, they're not.

Yeah. And that was actually part of what we built in the LLM gateway early on: the sort of last-chance hard stop. In the way that, if you go way afield on ChatGPT, you just get the inline 500 error, it doesn't even tell you that it can't help; it just craps out. We kind of built a couple of those circuit breakers, or the ability to put those circuit breakers in, and I don't believe we're using them for anything.

One last thing I want to get your thoughts on was AI fluency levels. You guys have a framework of user, advocate, builder, and native, and everyone goes through it, including Camila. I just think it's interesting. I think it's a model that other people are thinking about adopting, but they're worried about rolling it out—that everybody's going to be bad.

Well, and also, how do you have this in-house training course that you keep up to date? Just tell us more about it.

In operations, they're actually more ahead of even engineering on this front, as far as trying to create learning pathways for this. I think part of the reason why they're ahead of us is that, in operations, they're much more—they have to be able to operate training at scale. Training is a very big part of how people build aptitude around their job function within ops, whereas in EPD, a lot of it is getting hands-on building experience, going along, getting mentored, and getting code review.

But it's been really neat because I think we've created an environment by speaking openly about the transformation that we saw would happen in this industry toward AI, sort of displacing a lot of the operations and CX roles. We were just honest about it. In the same breath that we said, “Hey, a lot of these job responsibilities will go away,” we also said, “We don't anticipate that meaning that your job has to go away. It's just that your job has to change.”

The fluency framework, and then the training and support, and the positive culture where we celebrate people making progress, have been really helpful for avoiding a culture of fear, or a culture of, “Oh, you have to do this, or this is going to go in your performance evaluation.” It's not as rote as, “How much are you using AI, and is it enough?” I think we've built a pretty positively framed culture where we'll do spot bonuses for people who have particularly novel uses of AI in their day-to-day work.

In our company all-hands every 2 weeks, we'll do an AI spotlight, and it's very rarely somebody in EPD. For the most part, it's folks in GTM, ops, finance, and the people organization, showing off how they're building agents in ChatGPT or on Glean, or how they just found some new use case that they thought was helpful.

We're trying to create—I think, at the end of the day, we've hired a bunch of really smart people, and I have full confidence that this type of work is within the reach of anybody who's motivated to challenge themselves. And so, we've done that.

Then, in engineering, there's one other thing that I want to call out because I think this is kind of fun: we adapted our interview loop to be more AI-native, sort of agentic-coding-native. We had a coding and a system design question that we've basically revamped into a project where we'll give you a brief before you come on-site and then an additional spec when you start.

We expect you to use agentic coding to complete the task. In fact, it's kind of impossible to get all the way through it if you don't. We're evaluating your knowledge, watching how you work, evaluating whether you understand the code that's coming out, and probing you as you go.

What we did to bootstrap the process of getting all of our existing engineers familiar with agentic coding was that, as soon as we had the interview ready to ship, we said, “Everybody in engineering, including all the managers, is going to have to go through this interview.” We ran everybody internally through it.

It's one of those things where we didn't keep a score, and I don't have any data on who passed or failed or what they scored. But what we found is that, as people would take it, it would cause them to have moments of realization where it was like, “Oh, I can uplevel my skills,” or, “I want to be better at this.” So we're trying to find a variety of techniques to push the culture along.

As I reflect on the year, because this is the year where we really put all the effort into it, I'm really satisfied to see the extent to which everybody's leaning in on a daily basis. Going back to even when we were looking at our Cursor logs, I was shocked that the number-one user is an engineering manager in our org. That is super cool to me. It means folks have taken this to heart and found ways of doing their job differently.

I guess my closing question—or I guess a parting question—and this is broadening out from Brex—is that you interface with other engineering leaders all the time. Did we not cover anything that other CTOs are having top of mind today? What is their number-one problem?

The thing I find myself discussing with folks—and I don't want to shy away from scary topics—

In fact, we were just on one that was adjacent, which is: how do you evaluate somebody’s progression toward being more AI-native?

The cousin to that question is: Will we need as many people to operate our businesses? Are there layoffs coming? How are we thinking about headcount growth?

Junior versus senior.

Junior versus senior. Yes, exactly—level mix. I still have more questions than I have answers there. I think what has been really interesting is that I view agentic development as something that amplifies all the good just as much as it amplifies all the bad. It amplifies sloppiness, poor architectural thinking, and misunderstanding of the requirements. For all of the acceleration of good outcomes, it also accelerates bad outcomes.

What has been interesting is that, when you sum that all together, there’s less of an obvious capacity increase. It’s more nuanced than that. So I’m not looking at headcount planning as we think about it next year as something like, “Oh, well, because AI is giving us so much more leverage, we don’t need as many people.” We’ve actually—the thing I’m really proud of in my tenure as CTO is that we haven’t grown engineering at all. What we’ve done is grow the business significantly, but we’ve been able to build greater efficiencies in how we execute—how we think about building, how we roadmap, and what we choose to do and not to do—so that we’re able to serve significantly more customers with more lines of business without needing to grow engineering headcount.

I think that’s kind of the way that we’re going to continue on this road. I like having 300 engineers. I would love, love, love to just, a year from now, have 300 engineers, but we’re still 30%, 50%, or 100% more efficient. That is the thing that comes up with other engineering leaders. The other part of that conversation is how much AI is getting blamed for ordinary performance-oriented RIFs. If Microsoft is letting go of 4,000 people as a business—what, they have 150,000 employees, I believe—is that really AI causing that, or are they just using it as a way to avoid some harder performance-management decisions? I’m not entirely sure.

But I’m listening more than I’m speaking on this topic because every time I feel like I have a pretty firm point of view, some new anecdote or experience comes in that challenges or invalidates it.

Yeah. Well, I take these signals as it’s my job to go find people who think they have answers and surface them. You may or may not disagree, but at least you have something to use as a straw man in your work.

Exactly. Exactly. I think as an industry we’re just in the early innings of this transformation. So I’m looking forward to seeing—listening to this podcast episode a year from now—and seeing what we got right, what we got wrong, and what’s different, because so much changes quarter over quarter.

Yeah, I do think AI CoE is a very well-established pattern. I think an internal platform is a very well-established pattern, and this fluency thing is something that people are figuring out that I think you guys are ahead on. I’m happy to hear it. It’ll be my feedback.

Yeah.

Any final call to action for things that you want to buy? What should people build for you—problems you’re trying to solve that you would love people to reach out to help with?

The call that I’d make is for folks who are interested in multi-agent networks to get in touch with us, because I do feel like this is something where we’re innovating in service of our customers and where I feel like the frameworks, the tooling, and the research are there. There are actually quite a lot of interesting papers and things that we lean on. I would love to see more of that encoded in what’s available at large in the industry, because my intuition has been that trying to graft LLMs into deterministic workflows and DAGs is underselling the power that they have to actually plan and execute in a more sophisticated, fluid way.

I just want to see the industry lean in more on these agent-to-agent interactions.

Okay, so I’ll dive in a little bit here because I have a minor opinion. You keep using the word “networks.” Is that a reference to a specific paper, or is it your term for it?

It’s just our term, and I think that that is actually the term that Mastra uses as well. Initially, we used to call them agent runtimes internally, and then we switched to networks.

And then I think the other thing I wanted to get clarification on is: is it mostly a full agent talking with a full agent, or is there an orchestrator or boss agent talking to a sub-agent? I think that does matter for a subset of people who are building all these things, because when you say “multi-agent,” people don’t agree on what that means.

Yeah. So it’s a tree more than it is a graph. We have—

And when you say “network,” it feels more like a graph.

Yeah.

But it seems more directional, as a tree. There’s a hierarchy.

There’s a hierarchy. Yeah. But there are some violations of that. One of the interesting use cases—and this is where the power of having an assistant for every employee, plus having agents that run and embody members of the finance team, is really powerful—is that there’s this interesting use case that we brought to market.

One of the finance team agents that we launched is an audit agent. It embodies the work that a lot of larger finance teams will do to look for patterns of waste, fraud, or abuse, or systematic avoidance of policy that isn’t as obvious with a single expense. You can evaluate a single expense and the metadata around it to see if it’s within policy or not, but what if you start seeing an employee often make a large number of $74 transactions when receipts are required at $75?

This is a Brex customer’s SOP?

Exactly. Yep. What if you see certain things, like a fair number of DoorDash expenses during business hours from an individual on days that an office lunch is provided? Or maybe you see rideshare patterns where you have to look at broader context?

So we built this audit agent that can ingest your SOP and is always looking for potential violations. It’s extremely zealous: it wants to have a minimum number of false negatives. So it will raise a large number of potential violations, and then a separate review agent will apply the wisdom of whether this is important enough to follow up on, whether the dollar amount in question is high enough, and whether this user seems to have generally high compliance behavior.

It makes a judgment call about whether it’s worthy enough to take that violation and make it into a case. Then, once it’s made into a case, generally what happens is that you need to get more information from the individual. If humans were doing this, there would be some outsourced team looking for all the potential violations. Then you’d have a full-time employee on the finance team looking at all the violations and saying, “These are the ones that are important. We need to follow up on them.” Then they’d hand it off to somebody who would go and Slack that employee and say, “Hey, what’s going on here?”

What we have is the audit agent looking for violations, the review agent deciding whether they’re worthy enough to turn into a case, and then, from there, when the case is filed, that triggers an event to the Brex assistant for that employee. Any additional information about the business justification can be collected, or maybe the assistant already knows because, in its conversation history with the employee, it knew something about why this expense looked out of policy.

You start having the network become interesting when you have the finance team agents communicating with the assistants for various employees, and then behind that you have other sub-agents. You start seeing more of a graph emerge, but when you look at just what serves the employee, it looks more like a tree.

Amazing. Wow. I didn’t know you were going to go into that level of detail. Yeah, sorry about that.

No, no, no. I’m actually really glad I asked. That is very impressive, and I hope you do more content about that.

Yeah, absolutely. We’re really excited about it. I think it’s been good to finally figure out a use for agents and have the technology be as robust as it is to start realizing this vision, because it’s something that we dreamt of a couple of years ago. The technology just wasn’t there when we were trying to make a similar concept work with GPT-3.5. It was like, “No, we were hallucinating tool calls back in that day.”

Awesome, man. Thanks so much for joining us. This was fun.

I really enjoyed it. Happy holidays, guys. Thank you for having me.

Thank you.