The new OpenAI Agents Platform: CUA, Web Search, Responses API, Agents SDK!!

This is Alessio, partner and CTO at Decibel, and I'm joined by swyx, founder of Smol AI. Hi, welcome. Today we have a super-special episode because we're talking with our old friend Romain.

Thank you for having me.

And Nikunj, who, most famously—if anyone has ever tried to get access to anything on the API, Nikunj is the guy. I know your emails because I look forward to them.

Nice to meet all of you.

I think we're basically convening today to talk about the new API. Perhaps you guys want to kick off: What is OpenAI launching today?

I can kick it off. We're launching a bunch of new things today. We're going to launch 3 new built-in tools: the web search tool, an improved file search tool, and the computer-use tool.

The web search tool is basically ChatGPT Search, but available in the API. The file search tool lets you bring your data to OpenAI. You upload it, and we take care of parsing it, chunking it, embedding it, and making it searchable. We give you a ready vector store that you can use.

We're also launching our computer-use tool. This is the tool behind the Operator product in ChatGPT, and it's coming to developers today. To support all of these tools, we're launching a new API.

We launched Chat Completions in around March 2023, so it's been a while. We're looking for an update to support all the new things that the models can do, and we're launching this new API called the Responses API. It works with tools, and we think it'll be a great option for all the future agentic products that we build.

The last thing we're launching is the Agents SDK. We launched something called Swarm last year. It was an experimental SDK for people to do multi-agent orchestration and things like that. It was supposed to be educational and experimental, but people really loved it—they ate it up. So we thought, “All right, let's upgrade this thing and give it a new name.” We're calling it the Agents SDK, and it's going to have built-in tracing in the OpenAI dashboard. Lots of cool stuff is going out, so we're excited about it.

That's a lot. We said 2025 was the year of agents, so there you have it: a lot of new tools for developers to build these agents.

I guess we'll go one by one and leave the Agents SDK toward the end. For the Responses API, I think the primary concern people have—and something I voiced to you guys when I was talking with you in the planning process—is: Is Chat Completions going away? I wanted to let you respond to the concerns that people might have.

Chat Completions is definitely here to stay. It's a bare-metal API we've had for quite some time, with lots of tools built around it, so we want to make sure that it's maintained and that people can confidently keep building on it.

At the same time, it was optimized for a different world. It was optimized for a pre-multimodality world, and for single-turn interactions: text prompt in, text response out. Now, with these agentic workflows, we notice that developers and companies want to build longer-horizon tasks—things that require multiple turns to accomplish the task. Computer use is one example.

That's why the Responses API came to life: to support these new agentic workflows. Chat Completions is definitely here to stay. For the Assistants API, we've set a target sunset date of the first half of 2026.

In my mind, there was a poetic mirroring of the APIs with the models. I view this as the merging of the Assistants API and Chat Completions into one unified Responses API. It's kind of like how GPT and the old o-series models are also unifying.

That's exactly the right framing. We took the best of what we learned from the Assistants API, especially being able to access tools very conveniently, while simplifying the way you have to integrate. You no longer have to think about 6 different objects to access these tools. With the Responses API, you just make 1 API request and suddenly you can sweep in those tools.

Absolutely. I think we're going to make it really easy and straightforward for Assistants API users to migrate over to the Responses API without any loss of functionality or data.

Our plan is to add Assistant-like objects and thread-like objects that work well with the Responses API. We'll also add the Code Interpreter tool, which isn't launching today but will come soon. We'll add async mode to the Responses API, because that's another difference with Assistants, as well as webhooks and other features.

I think it's going to be a pretty smooth transition once we have all of that in place. We'll give folks a full year to migrate and help them through any issues they face. Overall, I feel like Assistants users are really going to benefit from this longer term, with this more flexible primitive.

How should people think about when to use each type of API? I know that, in the past, the Assistants API was more stateful—like long-running, many-tool-use, file-based things—and Chat Completions was more stateless, like a traditional Completion API. Is that still the mental model people should have, or should you, by default, always try to use the Responses API?

The Responses API is going to support everything that Chat Completions supports at launch, and then, over time, it's going to support everything that the Assistants API supports. It's going to be a pretty good fit for anyone starting out with OpenAI. They should be able to go straight to Responses.

The Responses API also has a stateless mode. You can pass in store: false, and that'll make the whole API stateless, just like Chat Completions. We're really trying to get this unification story in so people don't have to juggle multiple endpoints.

That being said, Chat Completions is the most widely adopted API. It's so popular, so we're still going to support it for years with new models and features. But if you're a new user—or an existing user who wants to tap into some of these built-in tools—you should feel totally fine migrating to Responses. You'll have more capabilities and better performance than with Chat Completions.

I think the messaging that resonated the most when I talked to you was that it's a strict superset. You should be able to do everything that you could do in Chat Completions and with Assistants.

The thing I assumed was that, because it's now stateful by default, you're actually storing the chat logs or the chat state. I thought you'd be charging me for it, so it was very surprising to me that you figured out how to make it free.

It's free. We store your state for 30 days, and you can turn it off. But yes, it's free.

The interesting thing about state is that, particularly for me, it makes debugging things and building things so much simpler. I can create a response object that's pretty complicated and part of this more complex application that I've built, then go into my dashboard and see exactly what happened.

Did I mess up my prompt? Did it not call one of these tools? Did I misconfigure one of the tools? The visual observability of everything that you're doing is so helpful. I'm excited about people trying that out and getting the benefits from it, too.

It's really nice to have. I'll just say that my friend Corey Quinn says, “Anything that can be used as a database will be used as a database,” so be prepared for some abuse.

That's a good one. Some people are very creative with the metadata, stuffing data into objects.

We do have metadata with Responses.

Let's get through all of these. For web search, when I first saw it, I thought you were going to just expose an API that returned a nice list of things. But the way it's named is GPT-4o Search Preview, so I'm guessing you're using basically the same model that's in ChatGPT Search, which is fine-tuned for search.

I'm guessing it's a different model from the base one, and the jump in performance is impressive. For example, on SimpleQA, GPT-4o has 38% accuracy, while GPT-4o Search Preview has 90%. We always talk about how tools are like models: the model is not everything, and the tools around it are just as important. Maybe give people a quick preview of the work that went into making this special.

Should I take that?

Firstly, we're launching web search in 2 ways. In the Responses API, which is our API for tools, it'll be available as a web search tool itself. You'll be able to go to Tools, turn on web search, and you're ready to go.

We still wanted to give Chat Completions users access to real-time information. In the Chat Completions API, which does not support built-in tools, we're launching direct access to the fine-tuned model that ChatGPT Search uses. We call it GPT-4o Search Preview.

As for how this model is built, our search research team has been working on this for a while.

Their main goal is to get a bunch of information from all of the data sources that we use to gather information for search, then pick the right things and cite them as accurately as possible. That’s what the search team has really focused on. They’ve done some pretty cool stuff. They use synthetic data techniques and model distillation to make these GPT-4o fine-tunes really good. The main thing is: can it remain factual? Can it answer questions based on what it retrieves, and can it cite it accurately? That’s what this fine-tuned model really excels at.

I’m super excited that it’s going to be directly available in Chat Completions, along with being available as a tool.

Just to clarify, if I’m using the Responses API, this is a tool, but if I’m using Chat Completions, I have to switch models. I cannot use o1 and call Search as a tool.

That’s right. Exactly. I think what’s really compelling, at least for me and my own uses of it so far, is that when you use web search as a tool, it combines nicely with every other tool and every other feature of the platform.

Think about this for a second. Imagine you have a Responses API call with the web search tool. Suddenly, you turn on function calling and also turn on, let’s say, Structured Outputs. Now you have the ability to structure any data from the web in real time in the JSON schema that you need for your application. It’s quite powerful when you start combining those features and tools together. It’s kind of like an API for the internet, almost. You get access to the precise schema you need for your app.

Then, just to wrap up on the infrastructure side of it, I read in the post that website publishers can choose to appear in the web search. Are publishers in it by default? How can we get Latent Space into the web search API?

I think we have some documentation around how website publishers can control what shows up in our web search tool, and you should be able to read that. I think we should be able to get Latent Space in for sure.

I compare this to a broader trend that I started covering last year of online LLMs. Actually, Perplexity was, I think, the first to offer an API connected to search, and then Gemini had the search-grounding API. I missed this in my original reading of the docs, but you even give citations with the exact subparagraph that matches, which I think is the standard nowadays.

My question is: how do we think about what a knowledge cutoff is for something like this? There’s no knowledge cutoff; it’s always live. But there’s a difference between what the model has internalized in its backpropagation and what it’s searching up with RAG.

I think it kind of depends on the use case and what you want to showcase as the source. For instance, take a company like Hebbia that has used this web search tool. For credit firms or law firms, they can combine public information from the internet with live sources and citations, which sometimes you do want to have access to as opposed to the internal knowledge.

But if you’re building something different, where you just want to have an assistant that relies on the deep knowledge that the model has, you may not need to have these direct citations. It depends on the use case a little bit, but there are many companies like Hebbia that will need access to these citations to precisely know where the information comes from.

For sure. One thing on the breadth: I think a lot of these Deep Research and open Deep Research implementations have this sort of hyperparameter about how deep they’re searching and how wide they’re searching. I don’t see that in the docs, but is that something we can tune? Is that something you recommend thinking about?

That’s super interesting. It’s definitely not a parameter today, but we should explore that. I imagine you would do it with the web search tool and the Responses API by having some form of agent orchestration, where you have a planning step and then each web search call explicitly goes a layer deeper and deeper and deeper. It’s not a parameter that’s available out of the box, but it’s a cool thing to think about.

The only guidance I’ll offer there is that a lot of these implementations offer top-K, which is top 10 or top 20, but you don’t really want that. You want some kind of similarity cutoff—a matching-score cutoff—because if there are only 5 things, 5 documents, that match, fine. If there are 500 that match, maybe that’s what I want, right?

But that might make my cost very unpredictable, because the cost is something like $30 per 1,000 queries, right?

I guess you could have some form of context budget, and then you’re like, go as deep as you can, pick the best stuff, and put it into X number of tokens. There could be some creative ways of managing cost. That’s a super interesting thing to explore.

Do you see people using File Search and the Search API together, where you can search and then store everything in the file so that next time I’m not paying for the search again? How should people balance that?

That’s actually a very interesting question. Let me first tell you about a really cool way I’ve seen people use File Search and web search together: they put their user preferences or memories in the vector store. A query comes in, you use the File Search tool to get someone’s reading preferences or fashion preferences, and then you search the web for information or products that they can buy related to those preferences. You then render something beautiful to show them: “Here are 5 things that you might be interested in.”

That’s how I’ve seen File Search and web search work together. By the way, that’s a single Responses API call, which is really cool. You just configure these things, go, boom, and everything happens. That’s how I’ve seen File Search and web search work together.

What you’re pointing out is interesting, and I’m sure developers will surprise us, as they always do, in terms of how they combine these tools and how they might use File Search as a way to have memory and preferences, as Nikunj says.

Zooming out, what I find compelling and powerful here is that when you have these neural nets with all of the knowledge they have today, plus real-time access to the internet for any kind of real-time information that you might need for your app, and File Search, where you can have a lot of company-private documents and private details, you combine those 3 and you have very compelling and precise answers for any kind of use case that your company or your product might want to enable.

There’s a difference between internal documents and the open web, right? You’re going to need both.

Exactly. Exactly. I never thought about it doing memory as well. I guess, again, anything that’s a database, you can store it, and they will use it as a database. That sounds awesome.

I think also you’ve been expanding File Search. You have more file types, query optimization, and custom reranking. It really seems like it’s been fleshed out. Obviously, I haven’t been paying a ton of attention to the File Search capability, but it sounds like your team has added a lot of features.

Metadata filtering was the main thing people were asking us for for a while, and that’s the one I’m super excited about. It’s just so critical once your vector store size goes over 5,000 or 10,000 records. You kind of need that. Metadata filtering is coming too.

For most companies, it’s also not a competency that you necessarily want to rebuild in-house. Thinking about embeddings and chunking and how all of that works, it sounds very complex for something that’s fairly obvious to ship for your users. Companies like Navan, for instance, were able to use File Search to take all of the FAQs and travel policies that you have and put that in the File Search tool. Then you don’t have to think about anything; your assistant naturally becomes much more aware of all of these policies from the files.

The question is that there’s a very vibrant RAG industry already, as you well know. There are many other vector databases and many other frameworks, probably if it’s an open-source stack. A lot of the AI engineers I talk to want to own this part of the stack, and it feels like: when should we DIY, and when should we just use whatever OpenAI offers?

If you’re doing something completely from scratch, you’re going to have more control, right? I’m super supportive of people trying to roll up their sleeves, build their custom chunking strategy and custom retrieval strategy, and all of that. Those are things that will be harder to do with OpenAI’s tools.

OpenAI has an out-of-the-box solution. We give you some knobs to customize things, but it’s more of a managed RAG service. My recommendation would be: start with the OpenAI thing and see if it meets your needs.

And over time, we're going to be adding more and more knobs to make it even more customizable. But if you want the completely custom thing, you want control over every single thing, then you'd probably want to go and hand-roll it using other solutions. So we're supportive of both; engineers should pick.

Yeah. And then we got computer use, which I think Operator was obviously one of the hot releases of the year. We're only 2 months in. Let's talk about that. And that also seems like a separate model that has been fine-tuned for Operator, with browser access.

Yeah, absolutely. The computer-use models are exciting. The cool thing about computer use is that we're so early. It's like the GPT-2 of computer use, or maybe the GPT-1 of computer use, right now.

But it is a separate model that the computer-use team has been working on. You send it screenshots, and it tells you what action to take. The outputs of it are almost always tool calls, and you're inputting screenshots based on whatever computer you're trying to operate.

Maybe zooming out for a second, because I'm sure your audience is super, super AI-native, obviously: What is computer use as a tool, right? And what's Operator? The idea for computer use is: How do we let developers also build agents that can complete tasks for users, but using a computer or a browser instead? How do you get that done? That's why we have this custom model optimized for computer use, which we use for Operator ourselves.

But the idea behind putting it as an API is that, imagine now you want to automate some tasks for your product or your own customers. Now you have the ability to spin up one of these agents that will look at the screen and act on the screen. That means the ability to click, scroll, type, and report back on the action. So that's what we mean by computer use, and wrapping it as a tool in the Responses API.

Now that also gives a hint at the multi-turn thing that we were hinting at earlier. Maybe one of these actions can take a couple of minutes to complete because there are maybe 20 steps to complete that task.

Do you think computer use can play Pokémon?

Interesting. I guess we tried it. I guess we should try it. There's a lot of interest. I think Pokémon really is a good agent benchmark, to be honest. It seems like Claude is running into a lot of trouble.

Sounds like we should make that a new eval.

Yeah, yeah, yeah.

And then one more thing before we move on to the Agents SDK. I know you have a hard stop. There are all these preview models, right? Like Search Preview and Computer Use Preview, and they seem to be fine-tunes of GPT-4o. I think the question is: Are they all going to be merged into the main branch, or are we basically always going to have subsets of these models?

Yeah, I think in the early days, research teams at OpenAI operate with fine-tuned models, and then once the thing gets more stable, we sort of merge it into the mainline. So that's definitely the vision: as we get more comfortable with them, learn about all the developer use cases, and do a good job at them, we'll make them part of the core models so that you don't have to deal with the bifurcation.

You should think of it this way: exactly what happened last year when we introduced vision capabilities. Vision capabilities were in a vision-preview model based off GPT-4, and vision capabilities now are obviously built into GPT-4. You can think about it the same way for the other modalities, like audio, and those kinds of models optimized for search and computer use.

Agents SDK—we have a few minutes left. So let's just assume that everyone has looked at Swarm. I think Swarm has really popularized the handoff technique, which I thought was really interesting for a multi-agent world. What is new with the SDK?

Yeah, for sure. We've basically added support for types. We've added support for guardrails, which is a very common pattern. In the guardrail example, you basically have 2 things happen in parallel. The guardrail can sort of block the execution. It's a type of optimistic generation that happens.

I think we've added support for tracing. So you can basically look at the traces that the Agents SDK creates in the OpenAI dashboard. We also made this pretty flexible, so you can pick any API from any provider that supports the Chat Completions API format. It supports the Responses API by default, but you can easily plug it into anyone that uses the Chat Completions API.

Similarly, on the tracing side, you can support multiple tracing providers. By default, it points to the OpenAI dashboard, but there are so many tracing companies out there, and we'll announce some partnerships on that front, too. So just adding lots of core features and making it more usable, but still centered around handoffs as the main concept.

And by the way, it's interesting, right? Swarm just came to life out of learning from customers directly that orchestrating agents in production was pretty hard. Simple ideas could quickly turn very complex: What are those guardrails? What are those handoffs? That came out of learning from customers and was initially shipped as a low-key experiment, I'd say.

But we were kind of taken by surprise at how much momentum there was around this concept. So we decided to learn from that and embrace it—to be like, “Okay, maybe we should just embrace that as a core primitive of the OpenAI platform.” That's kind of what led to the Agents SDK. And I think now, as Nikunj mentioned, it's adding all of these new capabilities to it, leveraging the handoffs that we had, but tracing also.

What's very compelling for developers is that, instead of having one agent to rule them all and stuffing a lot of tool calls in there that can be hard to monitor, now you have the tools you need to separate and spread the logic, right? You can have a triage agent that, based on an intent, goes to different kinds of agents. And then on the OpenAI dashboard, we're releasing a lot of new UI for logs as well.

So you can see all of the tracing UIs. Essentially, you'll be able to troubleshoot exactly what happened in that workflow when the triage agent did a handoff to a secondary agent and then a third, and see the tool calls, et cetera. So we think that the Agents SDK combined with the tracing UIs will definitely help users and developers build better agentic workflows.

And just before we wrap, are you thinking of connecting this with the RFT API? Because I know you already kind of store my text completions, and then I can do fine-tuning of that. Is that going to be similar for agents, where you're storing my traces and then helping me improve the agents?

Yeah, absolutely. You've got to tie the traces to the eval product so that you can generate good evals. Once you have good evaluators and tasks, you can use that to do reinforcement fine-tuning, and lots of details have to be figured out over here. But that's the vision, and I think we're going to go after it pretty hard and hope we can make this whole workflow a lot easier for developers.

Awesome. Thank you so much for the time. I'm sure you'll be busy on Twitter tomorrow with all the developer feedback.

Yeah, thank you so much for having us. As always, we can't wait to see what developers will build with these tools and how we can learn as quickly as we can from them to make them even better over time.

Awesome. Thank you guys. Thank you. Thank you both. Awesome.