DevDay 2025: Apps SDK, Agent Kit, MCP, Codex and why Prompting is More Important than Ever

Thank you for having us.

Yeah, it's always nice to be here.

It's such a nice thing. We've covered 3 of these DevDays now, and this is the first time it's been so well organized that we have our own little podcast studio in the DevDay venue. It's really nice that I actually get a chance to sit down with you guys. Thanks for taking the time.

I feel like DevDay is always a process. We've only had 3 of them, and we try to improve it every time. I know for a fact that we have this podcast studio this time because the podcast interviews, and the interviews with folks like yourselves last time, went really well, so we wanted to lean into it a little bit more. I'm glad that we were able to have this studio for you all.

We were kneeling on the ground interviewing Michelle last year.

I didn't know that. I just saw it in post-production.

We had to have people cordon off the area so they wouldn't walk in front of the cameras.

People would just come up. I'd be like, "We're recording here."

I guess if you guys have been to 3 DevDays, what stood out from today? What's your favorite part?

I feel like the vibes are just a lot more confident. You're obviously doing very well; you have the numbers to show it. Every year at DevDay, you report the number of developers. This year it's 4 million; I think last year was around 3 million. I have more questions about that kind of stuff, but there were also very interesting, high-confidence launches.

The community is clearly much more developed. There's a lot more to dive into across the API surface area of OpenAI than there was last year, at least in my mind. I don't know about you.

We were at the original DevDay, which was the DALL·E hack night at OpenAI in 2022, and I think Sam spoke to 30 people. It's crazy to see the—

Honestly, I think it's kind of similar to this podcast studio. We've had a number of DevDays now, and we're slowly figuring things out as a company over time, both from a product perspective and from how we want to present ourselves with DevDay.

At this third one, we've had a lot of feedback from people. The attendees will get an email with a chance to give feedback as well. We do read those, and we act on them. One of the things I really liked this year was all of the art installations and the little arcade games.

Yeah, the arcade games were so fun. I loved the theme of all the ASCII art throughout. This is my first San Francisco DevDay, but I've been to the Singapore one. That was actually my first week at OpenAI. I saw you there, so I was really in the deep end.

On a plane to Singapore.

Yeah.

Yeah, that's awesome. Congratulations on everything, and kudos to the organizing team. We should talk about some developer API stuff.

1. The Apps SDK Vision

We're going to cover a few things. You're not exactly working on the Apps SDK, but what should people generally take away from the Apps SDK launch? How do you internally view it?

The way that I think about it is that I view OpenAI, since the very beginning, as a company that has really valued opening up our technology and bringing it out to the rest of the world. One thing we talk about a lot internally is that our mission at OpenAI is, first, to build AGI, which we're trying to do, and second—potentially just as important—is to bring the benefits of that to the entire world.

One thing that we realized very early on is that, as a company, it's very difficult for us to bring it to every corner of the world. We need to rely on developers and other third parties to be able to do this. Greg talked about the start of the API and how that was formulated. That was part of this mentality: we needed to rely on developers, and we needed to open up our technology to the rest of the world so they could partake, for us to really fulfill our mission.

The API is a natural way of doing that. We literally expose API endpoints and tools for people to build things. Now that we have ChatGPT with 800 million weekly active users—I forgot the exact stat that we shared—and it's the 5th or 6th largest website in the world—

And the number 1 and number 2 most downloaded apps on the Apple App Store.

Oh, yeah, with Sora. But that moves around all the time, so it's hard to celebrate.

Just screenshot it when it's good.

Yeah, we definitely screenshotted and shared it earlier when it was good.

Going back to my main point, we've always engaged developers as a way for us to bring the benefits of AGI to the rest of the world. I view this as a natural extension of that. Candidly, we've been trying to do this a couple of times: last DevDay with GPTs, and 2 DevDays ago with—

Sorry, 2 DevDays ago with plugins, which was not tied to a DevDay.

I view this as a continuation of that process and of engaging deeply with developers and helping them benefit from some of the stuff that we have, which in this case is ChatGPT distribution.

On the Apps SDK, which is built on MCP, when did OpenAI become MCP-pilled? I'm sure internally you must have had design discussions about doing your own protocol. When did you buy into it, and how long ago was that?

I think it was in March, I want to say. It's hard for me to remember the exact—

March was the takeoff of MCP.

Okay, yeah. We built the Agents SDK, and we launched that alongside the Responses API in early March. As MCP was growing, that felt like a really natural protocol. We're building a new agentic API that can call tools and be much more powerful. MCP was the natural protocol that developers were already using to bring all the tools into their system.

In March, we added MCP to the Agents SDK first, and then soon after with our other—

There was definitely a moment, I think—a specific moment in a specific tweet. What I will say, though, is that credit goes to the team at Anthropic that created MCP. I really do think they treat it as an open protocol. We work very closely with David and the folks on the consortium, and they don't view it as something specific to Anthropic. They really view it as an open protocol.

The way in which you make changes feels very open. We actually have a member of our team, Nick Cooper, who sits on the steering committee for MCP as well. They are really treating it as something that's easy for us and other companies—everyone else—to embrace, which I think they should, because they do want it to be embraced by all. Because of that, I think it makes it a little bit easier for us to embrace it. Honestly, it's a great protocol. It's very general.

It's already solved. Why would you make it?

Yeah, it's very general. There's obviously still more to do with it, but it was very easy for us to integrate because of how streamlined and simple it was.

My final comment on the Apps SDK stuff, and then we'll move to AgentKit, is that, in the abstract, when you wireframe a website or an AI app, it used to be that the initial AI integration on a website would be the normal website, and then you'd have a little chatbot app.

Now it's kind of inverted: there's ChatGPT at the top layer, and then there's the website embedded inside it. That's an inversion I've been looking for for a little bit, and I think it's really well done. All the integrations and custom UI components that come up—you had Canva in the keynote, and it looks like Canva, but you can chat with it within all the context of your ChatGPT.

That is an experience I've never seen.

Yeah.

I think, going back to the iterative learning, we've learned a lot from plugins.

When we launched plugins, I remember one piece of feedback that we got. I don't know if people here really remember plugins—it was March 2023. One point of feedback was, “Oh, you told all these companies that they could integrate these plugins into ChatGPT, but they really didn't have that much control over exactly how they were used. It was really just a tool that the model could call, and you were really bound by ChatGPT.”

I think you can kind of see the evolution of our product with this. This time, we realized how important it was for companies and third-party developers to really own and steer the experience, to make it feel like themselves, and to help them preserve their own brand. I actually don't think we would have gotten that learning had we not had all these other steps beforehand.

Awesome. Christina, you were the star today on stage with the AgentKit demo. You had 8 minutes to build an agent, you had a minute to spare, and then you had some issues with the download with seconds left.

Honestly, I was like, “Let's do a little bit less testing.”

I was extremely stressed about the download thing. I was like, “If a UI bug is what takes the demo down, I'm going to be so sad.”

I think it was a fullscreen focus thing.

I heard the window wasn't in focus or something. Maybe you want to introduce AgentKit to the audience?

2. AgentKit Goes End To End

Yeah. We launched AgentKit today: a full set of solutions to build, deploy, and optimize agents. A lot of this comes from working with API customers and realizing how hard it actually is to build agents and then take them into production. It's hard to get that confidence and the iterative loop—writing prompts, optimizing them, and writing evals. All of that takes a lot of expertise.

Taking those learnings and packaging them into a set of tools makes it a lot easier and more intuitive to know what you need to do. There are a few different building blocks that can be used independently, but they're stronger together because you then get the whole end-to-end system. We're releasing that today for people to try out and see what they build.

I find it hard to hold all the building blocks in my head, but chronologically it's really interesting that you guys started out with the Agents SDK first, and then you have Agent Builder, a Connector Registry, ChatKit, and the evals stuff. Am I missing any major components? Those are the main moving parts, right?

Yeah, I think that's it. We also still have the RFT fine-tuning API, but we technically group it outside of the AgentKit umbrella.

Got it. Got it. Got it. It's weird how it develops, and it's now become a full agent platform, right? One thing that wasn't clear to me when I was looking at the demo was—it's very funny, because what you did on stage was build a live chat app for DevDay's website.

Did you get a chance to try it out?

Yeah, I tried it out. It was awesome. Actually, I wanted to ask: how do you deploy? I was like, “Where'd you click the merch?” Anyway, this is very close to home because I've done it for my conferences, and it's a very similar process. But I think what was not obvious is how much is going to be done inside Agent Builder. I see there are some actually very interesting nodes that you didn't get to talk about on stage, like human approval.

That's like a whole thing. You know, with transform and set state, there's a kind of Turing-complete machine in here.

I think, again, this is the first time that we're showing Agent Builder, so it's definitely the beginning of what we're building. Human approval is one of those use cases that we want to go pretty deep on. The node that I showed today is pretty simple—it's similar to what you'd see for MCP tools, approving that an action can take place.

But what we've seen with much more complex workflows from our users is that human-in-the-loop interaction is actually quite advanced. Sometimes these could be over the course of weeks, right? It's not just a simple approval of a tool. There's actual decision-making involved in it. As we work with those customers, we definitely want to continue to go deeper into those use cases, too.

What's the entry point? Are developers also supposed to come here and then do the code export, just to segment the use cases?

Yeah. I think the 2 reasons that you would come to Agent Builder are, 1, more as a playground to model and iterate on your systems, write your prompts, optimize them, and test them out. Then you can export it and run it in your own systems using the Agents SDK, using other models as well.

The second would be to get all of the benefits of us deploying that for you, too. You can use natural language to describe what type of agent you want to build, model it out, and bring in subject-matter experts so that you really have this canvas for iterating on it and getting feedback. You can build datasets and get feedback from those subject-matter experts as well, then deploy it all without needing to handle that on your own.

That's a lot of the philosophy around how we're building it with ChatKit as well. You can take pieces of it, and you can have a more advanced integration where it's much more customized. But you also get a really natural path of going live with really easy defaults.

Do you see it as a two-way thing? So I build here, I go to code, then maybe I make changes in code and then I bring those changes back to the Agent Builder?

That's definitely what we want to do. Maybe you could start off in code and bring it in. We'll also probably have the ability to run code in Agent Builder as well, so I think there's a lot of flexibility around that.

One thing I'd say, too, is that a lot of the demos that we showed today erred on the side of simplicity, just so that the audience could see it. But if you talk to a lot of these customers, they're building pretty complex systems. You have to zoom out on that canvas quite a bit to see the full flow.

For us, we were working with a lot of customers who are doing this, and if you turn that into an actual Agents SDK file, it's pretty long. We saw a lot of benefit from having the visual setup here, especially as the setup grows longer and longer. It would have been a little difficult to showcase this, but even in 8 minutes you can do it, with some of the presets that we have.

One of the things that we launched today, alongside the canvas, is a set of templates that we've gathered from our engineers who are working in the field with customers directly. These are common patterns from our own playbooks when we're working with customers on customer support and document discovery, and so we're publishing those as well.

Data enrichment, planning helper, customer service, structured data Q&A, document comparison—that's nice. Internal knowledge assistant.

I think we just plan to add more to those as we can build them out.

3. The Interoperability Question

I always wonder if there should be other agent builders. You're not the only agent builder, but obviously, by default, being OpenAI, you are a very significant one. Is there any interest in a protocol or interoperability between different open-source implementations of this kind of agent-builder pattern?

I think we've thought about it, especially around the Agents SDK. I would actually say maybe zooming out a bit more from just this: we're also sitting here and observing things being made over and over again. Even besides agent workflows, we're looking at what the industry is trying to do with Responses, like what we've done with the Responses API and stateful APIs.

Obviously, we were the first one to launch the Responses API, but a couple of other people have adopted it. I think Grok has it in their API. I think I saw LM Studio do something recently as well, but not everyone.

Unfortunately, I don't have a great answer today of yes or no, but we are assessing everything and trying to see what could become a standard. There has been a lot of value with MCP, and hopefully with our commerce protocol as well, ACP. I definitely did not forget the name.

Even thinking about what we want to do with agents and agent workflows, the portability story around that would be great. The portability of the Responses API would be great, too, if that could be a standard or something, so developers don't need to build 3 different stateful API integrations if they want to use different models.

Yeah, and I think that's one of the—it's not exactly a protocol, but one of the things that we launched today with evals, too, is the ability to use third-party models and bring that into one place. I think you can definitely see where the ecosystem is at, which is using multiple models.

Third-party models, as in non-OpenAI models?

Yeah. Yeah. It'll work with evals starting today.

Yeah.

Okay, got it. We have a really cool setup with OpenRouter, where we're working with them, and then you can bring your OpenRouter setup. With that, you can actually write your evals using our Datasets tool, or use our Datasets tool to create a bunch of evals, and you'd be able to hit a bunch of different model providers. You can take your pick from wherever, even open-source ones on Together, and see the results in our product.

Yeah, that's awesome. Speaking more about evals, I think I saw somewhere in the release docs that you had to expand the evals product a little bit to allow for agent evals. Maybe you can talk about what you had to do there.

Yeah.

Yeah. I was going to say, agent evals are still a work in progress. I think we've made maybe 10% of the progress that we need here. For example, I think we could still do a lot more around multimodal evals.

But the main progress that we made this time was allowing you to take traces. The Agents SDK has a really nice traces feature where, if you define things, you can have a really long trace, allowing you to use that in the evals product and be able to grade it in some way, shape, or form over the entirety of what it's supposed to be doing.

I think it's step 1. It's good to be able to do this, but I think our roadmap from here on out is to really allow you to break down the different parts of the trace, and allow you to evaluate, measure, and optimize each of those as well. A lot of the time, this will involve a human in the loop, which is why we have the human-in-the-loop component here, too.

But if you look at our evals product over the last year, it's been very simple. It's been much more geared toward this simple prompt-completion setup. Obviously, as we see people doing these longer agentic traces, how do you even evaluate a 20-minute task correctly? It's a really hard problem. We're trying to set up our evals product to move in that way, to help you not only evaluate the overall trajectory but also individual parts of it.

Yeah, I mean, the magic keyword is rubrics, right? Everyone wants LLM-as-a-judge rubrics.

Yeah.

Obviously, that's where this will go. Okay, great. The other thing I see the developer community very excited about online is automated prompt optimization, which is kind of eval in the loop with prompts. Where's the thinking there? Where are things going?

Yeah, so we have automated prompt optimization, but again, I think this is an area that we definitely want to invest more in. We did a pretty big launch of this when we launched GPT-5, actually, because we saw that it was pretty difficult, as new models come out, to learn all the quirks about a new model. We have a big prompting guide for every model that we launch, and I think building out a system to make that a lot easier is important.

We definitely want to tie that in completely with evals. We should be able to improve your prompts over time and improve your agents over time as well, if they're made in the Agent Builder, based on the evals that you've set up. I think we see this as a pretty core part of the platform: suggested improvements to the things that you're building.

I actually think it's a really cool time right now in prompt optimization. I'm sure you guys are seeing this, too. Not only are there a lot of products gearing around this, which is kind of what we're thinking about, but I also think there's a lot of interesting research around this. GEPA, with the Databricks folks, is doing really cool stuff around this.

We're obviously not doing any of the cool GEPA optimization right now in our product, but we'd love to do that soon. It's also an active research area, so whatever Matei and the Databricks folks might think about next, we might think about internally as well. Whatever new prompt optimization techniques come out, we'd love to be able to have that in our product as well.

It's interesting because it's coming at a time when people are realizing that prompting—I feel like 2 years ago, people were like, “At some point, prompting is going to be dead.” No.

It's gone up.

Yeah. Yeah. Yeah. If anything, it has become more and more entrenched.

And I think there's this interesting trend where it's becoming more and more important, and there's also interesting, cool work being done to further entrench prompt optimization. That's why I just think it's a very fascinating area to follow right now, and also an area where I think a lot of us were wrong 2 years ago, because if anything, it's only gotten more important.

I would say that someone who used to work at OpenAI and is now an ML scientist would call this zero-gradient fine-tuning, or zero-gradient updating, because you're just tweaking the prompt. But it is so much prompting that you end up with a different model at the end of it.

There are a lot of things that make it more practical, too. Even from our perspective, we have a fine-tuning API, and it is extremely difficult for us to run and serve all of these different snapshots. LoRA is great. Thinking Machines just published something, and John Schulman just had a cool blog post about this.

It is pretty difficult for us to manage all of these different snapshots. If there were a way to hill-climb and do this zero-gradient optimization via prompts, I'm all for it. I think developers should be all for it, because you get all these gains without having to do any of the fancy fine-tuning work.

Since you are part of the API team—you lead the API team—and since you mentioned Thinking Machines, I have to throw a cheeky one in there. What do you think about the Tinker API?

It's a good one. It's actually funny: when it launched, I DM'd John Schulman, and I was like, “Really? Wow. We finally launched it.”

Because you used to work with him.

Yeah. So, right when I joined OpenAI, this had actually been a passion project of John's. He's been talking about doing something in this shape for a while, which is a truly low-level research fine-tuning library. We talked about it quite a bit when he was at OpenAI as well.

It's actually funny: I talked to one of my friends, who said that when John was at Anthropic, he also worked on the idea for a bit. I think now—

He's a man on a mission.

Yeah. John's so great in this regard. He's purely interested in the impact of this, because, first, it's a really cool problem, and second, it also empowers builders and researchers. You saw all the researchers who expressed all this love for Tinker, because it is a really great product. I'm just really happy to see that they shipped it, and I think he was really happy to get it out there in the world as well.

This is probably a digression, but as someone passionate about API design, it's weird that it took this long to find a good fine-tuning API abstraction. That's effectively all he wanted. He was like, “Guys, I don't want to worry about all the infrastructure. I'm a researcher. I just want these 4 functions.” It's kind of interesting.

Yeah. Yeah.

Cool.

Before the OpenAI comms team barges into the room.

I know.

What feedback do you want from people on the Agent Builder? For example, the thing I was surprised by was that the if-else blocks weren't in natural language and used the Common Expression Language. I'm sure that's already on your roadmap. What are other things where you're kind of at a fork and would love more input?

I think one of the things that we spent a lot of time discussing was whether we want more deterministic workflows or more LLM-driven workflows. Getting feedback on that would honestly be helpful, as would having people model existing workflows.

A lot of what we did was work with our team, especially engineers who are working with customers, to model the workflows that already exist in Agent Builder and identify what gaps exist. What types of nodes are really common, and how can we add those in? I think that would be the most helpful feedback to get back.

As we expand beyond chat-based use cases—the initial deployment for Agent Builder is through ChatKit—we plan on releasing more standalone workflow runs as well, along with the types of tasks that people would like to use in that type of API.

More modalities, for example.

Yeah, I think for sure more modalities. Voice is already something that a lot of people have talked to us about, even today at Dev Day. So, modalities for sure, but also more of the logical nodes—what can't be expressed today.

Yeah. Well, you're building a language, right? You have Common Expression Language, which I had never heard of prior to this. I thought this was Python, this was JavaScript, and then there was a whole language in there. Was that a big decision for you guys?

I think that was more just a way that we thought we could represent a mix of variables and conditional statements.

The other thing I'll also mention is that there's a trope in developer tooling: anything that can store state will eventually be used as a database, including DNS. Be prepared for your state store to become a database. I don't know if there are any limits on that, because people will be using it.

It's actually funny. I'd heard this quote before, and there's definitely some truth to it. I don't know if our stateful APIs have become a database just quite yet, but who knows?

Well, you charge for it. You charge for Assistants—the storage, right?

The storage, right? So there's some limit on that.

Yeah, but it's very cheap. I remember we priced it like—

I think if you wanted to dump all your data somewhere, transforming it all into this shape...

It's useful. It's easy—the best place to put it—but, yeah. Also, please don't do this, because I think it'll put quite a bit of strain on Vendo and our infra team and what we try and do.

4. The Connector Tradeoffs

How do you think about the MCP side? You have OpenAI first-party connectors, third-party preferred servers, I guess you would call them, and then you have open-ended ones. Do you see that part of registry-like functionality expanding, or do you see most of it being user-driven? OAuth is, like, the biggest thing: if you add Gmail, Calendar, and Drive, you have to authenticate each of them separately. There's not a canonical approach. What's the thinking there?

Yeah, I think definitely for the registry, that's why we want to make it a lot easier for companies to manage what their developers have access to and manage the configurations around it. In terms of first-party versus third-party, we want to support both of those. We have some direct integrations, and anyone can create MCP servers. I think we want to make it a lot easier to establish private links for companies to use those internally. So, I'm really excited about that ecosystem growing.

I think one of the coolest things I've observed, too, is that I actually think we, as an industry, are still trying to figure out the ideal shape of connectors. Part of why I think the first-party connectors exist, too, is that we end up storing quite a bit of state. It's a lot of work for us, but by having a lot of state on our side—we call them sync connectors—we can actually end up doing a lot more creative stuff on our side when you're chatting using these connectors to boost the quality of how you're using it, right? If you have all the data there, you can do all this reranking. We can put it in a vector store if you want; you can put it anywhere else.

There are some inherent trade-offs here. You put in a lot of work to get these first-party connectors working, but because you have the data, you can do a lot more and get higher quality. But then the question is, oh my God, there's such a long tail of other things, which is where MCP and third-party connectors come in. Then you have the trade-off that you're beholden to the API shape of the MCP creator. It might actually work well; it might not work well with the models. And what happens if it doesn't work well? Then you're kind of at the mercy of this.

MCP, by the way, is really great because it already does some layer of standardization, but my sense is there's still going to be more evolution here. I think we want to support both of them because we see value in both. Right now, especially working with developers, we want to have all options on the table here, but it will be interesting to see how this evolves over time.

Yeah, when I saw, 3 or 4 months ago, that you launched the form for expressing interest in Sign in with ChatGPT, I thought that was kind of the vision: I log in, I have the MCPs tied in, and then I sign in with ChatGPT somewhere and can run these workflows in the app where I'm logging in. Sam said in an interview that ChatGPT is like your personal assistant, so I think this is a great step in that direction. There's a lot more to go in that direction.

But so far, no plan on ChatGPT as an OpenID provider, right? That's a different role in the auth ecosystem.

Yeah, the direct answer is no plans right now, of course. But I actually think we currently have some version of this, which is our partnership with Apple. With Apple, you can actually sign in to your ChatGPT account, and some of that identity carries with you into your iOS experience with Siri.

If you've used the Siri integration—I actually use it quite a bit—when you sign in to your account, the Siri integration will use your subscription status to decide what type of model to use when it passes things over to ChatGPT. If you're just a free user, you get the free model, but if you're a Plus or Pro subscriber, you get routed to GPT-5, which I think is what they have.

I think we also recently announced the partnership with Kakao.

Oh, yeah. Kakao is another one.

Yeah. It's a similar thing where you can sign in with ChatGPT. Kakao is one of the largest messenger apps in Korea, and you can interact with Kakao directly there.

Yeah, I mean, Sam's been talking about it for a while. It's a very compelling vision. We obviously want to be very thoughtful with how we do it.

You know, now you have a social network, you have a developer platform—very, very valuable. Exactly. On the other side of auth, something I was really interested to look at, and couldn't get a straight answer on, is whether there's some form of bring-your-own-key for AgentKit. When I expose it to the wider world, obviously by default I'm paying for all the inference, but it'd be nice for that to have a limit, and then if you want more, you can bring your own key.

Yeah, we don't have something like that yet, but I think it's definitely an interesting area to explore.

Yeah, it doesn't do it out of the box today, but developers have been asking about it forever.

It's a really cool concept because, as a developer—especially an indie developer—you don't need to bear the burden of inference.

Yeah, I think when you get into the business of agent builders that are publicly exposed, where you have an allowlist of domains, it rhymes with this exact pattern: someone has to bear the cost. Sometimes you want to mess around with the different levels of responsibility.

Yeah, I will say in general, if you look at our roadmap, we engage a lot with developers. We hear what the pain points are, and we try to build things that address them. Ideally, we're prioritizing in a way that's helpful.

We've definitely heard from a good number of developers that the cost—or all of the copy-paste-your-key solutions right now, which are huge security hazards—is a problem because developers don't want to bear the burden of inference. Hopefully we make the cost cheaper, so as the models keep getting cheaper, hopefully that helps. But what we realize is that as we make it cheaper, the demand for that goes up even more, and you end up still spending quite a bit. So we definitely hear this from a lot of developers, and it's definitely top of mind.

5. AgentKit Reaches Users

Do you see this as mostly an internal tools platform, though? To me, you've been doing a big push on the more forward-deployed engineering side. It's almost like, hey, we needed to build this for ourselves as we sell into these enterprises; we might as well open it up to everybody. What drives building these tools? Do you think of people building tools to then expose, or is it mostly on the internal side?

Yeah, I mean, our first deployment is ChatKit, which is intended to be for external users. But one of the things that we also saw a lot as we were working with customers is that a lot of companies have built some version of an agent builder internally to manage prompts and templates that they're sharing across the different developers they have, and maybe across different product areas.

We were seeing that over and over again as well, and really wanted to build a platform so that this is not an area that every company needs to invest in and rebuild from scratch. They can have a place where they can manage these templates and prompts and really focus on the parts of agent building that are more unique to their business.

It is interesting, too, from a deployment perspective, that it has spanned both internal and external use cases. These internal platforms people use for data processing or something are an internal use case, but if you saw some of the demos today, there have been a huge number of companies trying to do this for external-facing use cases as well.

Customer service is one—the Ramp use case.

We use this internally and externally. Our customer support at help.com is already powered by AgentKit, and then there are various internal use cases as well.

One of the things that I actually think the team has done a really great job of—the team members Tyler, David, and Guan—they built the ChatKit components to be very consumer-grade and very polished. You look at it and there’s a whole grid of the different widgets and things that you could create there. People see it as these very polished, consumer-grade, ready-to-use, external-facing things, versus—you know, you think of internal tools and the UI is always the last thing that people care about.

But we really pushed the team, and I think they did a great job of making the ChatKit experience really, really consumer-grade. It should feel almost like ChatGPT, with buttery-smooth animations, really responsive designs, and all of that.

Yeah, I think your point on widgets definitely resonates, right? ChatKit handles the chat UX, but we’re also building really visual ways for you to represent every action that you want to take. That is definitely very highly polished.

Yeah. And when working with customers, those have been the most helpful customers for us to work with, because when Ramp is thinking about what they want to publicly present to people, they have a pretty high bar, as they should, as well as all the other customers that have been iterating on it. That kind of feedback from our customers has really helped us uplevel the general product quality of the launch that we’ve had today as well.

Would you ever open source ChatKit?

Talked about it.

Uh-huh.

We’ve talked about it. There are a bunch of trade-offs.

I think so. ChatKit itself is an embeddable iframe.

An iframe, I thought, right? And so that helps us keep it evergreen, right? If you are using ChatKit and we come up with a new—

I don’t know, a new model that reasons in a different way, right, or new modalities that you don’t actually need to rebuild and pull in new components to use it in the front end.

Yeah, a more evergreen experience that is pretty opinionated.

There would be no point in being open source.

You want that, then you don’t get the benefits of it.

You know, being Stripe alums, like Stripe Checkout—it’s auto-optimized for you to—

So I’m not a Stripe alum, but Christina is.

And the team actually is the team that built—

Stripe, yeah. It’s very similar philosophically, right? Stripe can build Elements and Checkout, and not every business needs to rebuild the pieces that are really common. I think we see the same with chat. We see chat being built over and over again, especially as we come up with new modalities, like reasoning and everything. It’s not really something that’s easy to keep up to date, and so we should just do that and leave the hard parts of building agents to—

To the developers.

Does it feel—I mean, I know WordPress has a bad connotation in a lot of circles—but to me, it almost feels like the WordPress equivalent of chat. It’s like, hey, this is a drop-in thing, and then you have all these different widgets. Do you see the widget becoming a big developer ecosystem where people share widgets? Is that kind of a first-party thing? And what’s the MCP versus—

Widget forest?

No, exactly. I mean, it seems great for people who are in between being technical and not really being technical enough.

Yeah.

Yeah. I think that’s a big part of building widgets, right? It’s already in a language that is very consumer-friendly. You can already use AI in our widget builder to create those widgets, and they look pretty good. I don’t know if you’ve gotten a chance to try that out yet, but definitely see if you haven’t tried out the Widget Studio and the demo apps as well.

You got a custom domain like widget.studio, which is cool.

I actually don’t know how we got that.

Yeah, everything’s in ChatKit Studio, and then we have the playground there so you can try out what it would look like with all the customizations. We have ChatKit.world, which is a fun site we built.

I was spinning the globe for a while this morning. It was a widget spinner.

Kasia also uploaded some of her solar system stuff and all the demos as well.

Yeah. And then that’s where the widget builder is.

Yeah. So it’s really come together. It’s taken almost more than a year to build all this stuff, but it’s coming together. It’s really interesting.

Yeah. It’s something that we like—

You definitely planned all this up front.

Oh, yeah. We have the master plan from 3 years ago. No, but I think especially on this stuff, there was an arc of a general platform that we did want to build around, and it takes a while to build these things. Obviously, Codex has helped speed it up quite a bit now, but it does seem great to start having all the pieces fit together.

You saw we launched evals, and we had the fine-tuning API for a while. We laid all the groundwork for some of this stuff over the last year, and we’re hoping that we can eventually make it into this full-featured platform that’s helpful for people.

6. Codex Changes The Workflow

Since you mentioned Codex, maybe a quick tip from each of you on Codex power-user tools or tips?

There’s actually a funny one that one of the new grads has taught our team in general. I think this is a point about how new grads and the younger generation are actually more AI-native. One of them is to really lean in—to push yourself to trust the model to do more and more.

I feel like the way I was using Codex—for me, it’s mostly for my personal projects—they don’t let me touch the code anymore. I would give it small tasks, so I wasn’t really trusting it. I viewed it as an intern that I really didn’t trust. What a lot of the interns would do is just go full YOLO mode and trust it to write the whole feature. It doesn’t work sometimes, but I don’t know, 30% or 40% of the time it just one-shots it.

I actually haven’t tried this with GPT-5-Codex. I bet it probably one-shots it even more. One tip that I’m starting to have to undo and relearn is to really lean into the agentic component of it and let the model rip, and kind of trust it, because a lot of times it can actually do stuff that surprises me. Then I have to readjust my priors.

Before, I felt like I was in this safe space of giving this thing a tiny bit of rope, and because of that I was limiting myself in how effective I could be.

Sure, but okay, is there an etiquette around submitting effectively vibe-coded PRs that someone else now has to review? It can be—

Codex does reviews now. It actually reviews itself.

Does Codex approve its own PRs a lot more than humans?

It doesn’t approve them, but I was going to say that the Codex PR reviews are actually one of the things that my team very much relies on. I think they’re very high-quality reviews. On the Codex PR side, for the visual agents builder, we only started that probably less than 2 months ago, and that wouldn’t be possible without Codex.

So I think there's definitely a lot of use of Codex internally, and it keeps getting better and better. People are just finding they can rely on it more and more. It's not totally vibe-coded. It's still checked and edited, but definitely as a kicking-off point.

I've heard of people on my team who, on their way to work, kick off 5 Codex tasks because the bus takes 30 minutes. They get to the office, and it helps them orient themselves for the day. They're like, "Okay, now I know the files. I have a rough sense. Maybe I don't even take that PR, and I actually still code it." But it helps you context-switch so much faster, too, and orient yourself in a codebase.

There are so many meetings nowadays where I have 1-on-1s with engineers, and I walk into the room and they're like, "Wait, wait, wait. Give me a second. I've got to kick off my Codex thing." I'm like, "Oh, sorry."

We're about to sync our notes, right? You're like, "Let me—"

And they're like, "Finally. Okay, now we can start our 1-on-1 because now it's great."

Yeah. Cool. We're almost out of time. I wanted to leave a little bit of time for you to shout out the service health dashboard because I know you're passionate about it.

Oh, yeah.

Well, tell people what it is and why it matters.

7. The Service Health Dashboard

Yeah. So this is a launch that we actually didn't get any stage time for today, but it was something I'm really excited about. We launched this thing called the Service Health Dashboard. You can now go into your usage or settings in your account and see the health of your integration with our OpenAI API.

This is scoped to your own org. Basically, if you have an integration that's running with us, doing a bunch of tokens per minute or a bunch of queries, it's now tracking each of those responses, looking at your token velocity, the TPM that you're getting, the throughput, as well as the response codes. So you can see a real-time personal SLO for your integration.

The reason why I care a lot about this is that, obviously, over the last year, we've spent a lot of time thinking about reliability. We had that really bad outage last December—the longest 3 or 4 hours of my life—and then I had to talk to a bunch of customers.

We haven't had one that bad since, knock on wood. We've done a bunch of work. We have an infra team led by Venkat, and they've been working with Janna on our team. They've just been doing so much good work to get reliability better.

So we actually, again, knock on wood, think we've got reliability in a spot where we're comfortable putting this out there and letting people actually see their SLO. Hopefully, it's 3 nines, 4, soon to be 5 nines. The reason why I care a lot about it is because we spent so much time on it, and we feel confident enough to have it behind a product now.

5 nines is like 2 minutes of outage or something.

Yeah. We're working to get to 5 nines.

What does an extra nine take?

It's exponentially more work. In the last couple of years, we were talking about hitting 3 nines, then hitting 3 and a half nines, and then hitting 4 nines. But yeah, it's exponentially more work. I could go on for a while about the different topics, but—

We'll have to do that in a follow-up. I mean, that's all the engineering side, right?

Yes.

You're serving 6 billion tokens per minute.

We actually zoomed past that. That's outdated.

Yeah, it's been crazy, the growth that we've seen. Um, awesome. I know we're out of time. It's been a long day for both of you, so we'll let you go. Thank you both for joining us.

Thanks for having us.

Thank you.