打造金融服务版 ChatGPT——OpenAI 金融服务负责人 Joseph Kim

I really do think there’s going to be a $1 billion business built on top of a ChatGPT plugin. But 95% of what we built was something a startup or another company could absolutely do. At OpenAI, we ran into problems with pre-training and therefore had difficulty scaling the model. In 2026, we’ve been able to solve many of these issues.

Today we have Joseph Kim, one of the heads of ChatGPT Finance at OpenAI, and our co-host and head of editorial from Latent Space, Richard, in the studio. It’s a special episode. We’re here in New York at AI Finance. Joseph just gave a great keynote. I don’t think we want to repeat most of the information, so—

That makes sense.

Honestly, go watch it, but give us 30 seconds on what you talked about, and then we’ll leave the breadcrumbs. You guys can go watch the keynote.

Sure. I can give a quick 30-second intro to what we just talked about. We just released a new product, ChatGPT for financial services. It’s an enterprise-focused product that’s trying to bring the power of ChatGPT—the tools and things you like to use in a consumer product—and add the enterprise layer. That means firm-specific templates and sources like Daloop and S&P. We’ve been working with our design partners, Morgan Stanley and Evercore, and the reception has been incredible.

That was the first half of the talk. The second half was much more forward-looking. I talked about some of the trend lines we’re seeing in model improvements, projected them into the future, and considered what these models might be able to do by mid-2027 or the end of 2027. What does that mean for people building products on top of those models? How can you start planning for the intelligence that’s coming in just a matter of months?

I think it’s a fun time because you actually come from the research side of things. You now lead a lot of the product and engineering, but tell us what it takes to make a financial-specific product. You mentioned in the keynote that there was an uptick in users—

Yeah.

—when you launched ChatGPT Work, and Codex started to become available to the masses. People started using it for finance, and now you’re shipping your first finance product.

Yeah.

What is financial reasoning data like? What goes into finance-specific models?

It’s a great question. All of the training data we use for financial reasoning and analysis goes upstream into what we call the Big Run, which is the main run that becomes something like 6-Ash or 6-1-Ash, et cetera. Even though we’re adding a lot of financial data to the model, it’s going into the main model, so it’s available to people who want to build on top of it. That’s very important to us. At OpenAI, we want that intelligence to be accessible not only through the products we build in-house, but also to people who want to build startups and businesses on top of it, as well as financial firms that want to integrate it directly into their companies.

I think you touched on product and research, and that’s core to what we do at OpenAI. Internally, we call ourselves a research and deployment company. We structure that order intentionally: research comes first, and deployment comes second. As we think about changing the way people do knowledge work, we work in that order. First, we think about model capabilities, where they’re headed, and what new things they can enable. Then we think deeply about the product features and surfaces we should build to make that model intelligence accessible to more people.

You alluded to a slide in my keynote where we showed Codex adoption within OpenAI. It’s funny because you can see the software engineering adoption curve: it’s steadily up and to the right. People are picking it up and incorporating it into their daily routines. But if you look at finance, legal, and sales, adoption is pretty much at zero, and then suddenly it jumps to 90%. If you look at what happened around the time it jumped, we had released the Codex desktop app in February. That’s when adoption jumped. I think it shows the importance of developing those things together. The purpose of the product is to make that model intelligence accessible to as many users as possible.

In that uptick in using Codex for financial applications, what did you learn that you then applied to the post-training of financial data?

It’s a great question. I can talk a little bit about how we went about building ChatGPT for financial services. It was very unique. We haven’t built a lot of first-party products. Codex was our biggest and main one, and this was the first time we developed a product where the product, the model harness, and the research were all under one roof. That gave us a lot of flexibility in how we went about building things.

1. Building Around Industry Reality

The biggest takeaway from working on this for the last 6 months is that, in enterprise, you have to deal with the reality of the industry you’re in and build around that. That’s easy to ignore. If you think about it from first principles, you can reach a conclusion that makes sense and defend it with logic and reason. But if you ignore the state of the world when you start building, and the forces at play within the industry, you’re not going to build the right product. There are competitive forces, and people have their own objectives. We thought about that a lot with ChatGPT for financial services.

I’ll give you one example: data. Data was a huge part of how we thought about building the product. It was a little tough to defend from first principles when I was working on this product. You might say, “I want to bring quarterly earnings transcripts into the product and bundle them.” The first thing people will say is, “Look, there’s an MCP connector in the store. What incremental value are you providing by bundling that in?” On paper, it looks bad. They might already have a license for it, so they might feel like they’re paying twice. They might also have an MCP connector. So why are you building this?

The reality is that we tested all these MCPs, and the error rates were really high. Some of the MCPs we connected to, including some big names, had error rates of more than 40%. MCPs are one of those things that look great on paper, but you have to look at the data and see what’s actually going on. We found that the data experience was really bad. The only way to fix it at scale was to bring the data in-house and control the experience end to end. The extra money you pay to bundle that data is worth it. That was a product decision we had to make that was very industry-specific, and it’s resonating with customers. Customers love it. The number one thing they call out is the data, so it worked.

So you had good success with this.

Yeah.

Tell us about the technical side of building a harness adaptation for a specific service. You changed Codex, did post-training, and did whatever you did to make—

Yeah.

—ChatGPT for financial services.

Yes.

If someone were to do the same for—

Yeah.

—another domain, say legal or sales, whatever—

Yeah.

—what are the best practices? What did you learn? What goes into taking something off the shelf and actually making a product around it?

2. ChatGPT Plugins Become Products

I like the question a lot. I alluded to this in my keynote as well, but the interesting thing is that it’s fun to talk about the post-training stuff because it’s unique to what we do, and it gets people excited. But 95% of what we built was something a startup or another company could absolutely do.

Mm.

And we didn’t really have to leverage our ability to make changes in the model to build a fantastic product, right? I think it’s a great call to mention that piece. What I would say is, a few things. One is, most of the product we built was built on top of the Codex—or I should say ChatGPT now—plugin architecture and plugin primitives, and that’s something we’ve been investing a lot of time and energy in. OpenAI is an interesting company where we face the same problems internally that a startup would face if it were to build on top of OpenAI.

In a lot of ways, my team is like a startup building on top of OpenAI technology, right? Even internally, we need to find scalable ways for teams like mine to stand on the shoulders of giants and build on top of Codex. Codex is fantastic. It’s got great memory and good skills. It’s persistent.

Yes.

There are a lot of things that are wonderful about Codex, but the thesis of my team was that Codex will get you about 80% of the way to a product that would be successful in the market, but the 20% is missing. We need to find some way for my team to build that 20% and actually go in, because the other half of the thesis was that if you just took Codex and launched it as a financial product, it would fail. That 20% is pivotal.

We built on top of the plugin architecture, using the same primitives that are available to any of the builders who want to build on top of ChatGPT. I had a line in my presentation, and I really do think there’s going to be a billion-dollar business built on top of a ChatGPT plugin. You’re going to be able to leverage all of that horizontal functionality and just add your secret sauce on top. That secret sauce might be data, UI changes, or an integration. It’s really valuable, and I think there’s a billion dollars’ worth of value to be built there.

Mm-hmm.

It’s an interesting time for the industry because orchestration is becoming a lot more automated, with teams of agents going out and doing things.

Yeah.

But for finance, obviously, there are a lot of risks involved in that. Can you talk a bit about the safety you’ve baked into the product?

Yeah. Safety is super, super important. I would say it’s only going to get more important because agents are already getting more autonomous and more independent, and I think that trend is expected to continue into 2027 and 2028.

Mm.

For the product that we built and launched last month, there were a bunch of paradigms that already existed as it pertains to safety: being thoughtful about auditing the work and making sure the model isn’t making a mistake that’s going to impact business outcomes. We were able to lean heavily into those. The existing product that we launched is a productivity tool for knowledge workers in the space.

Mm.

The idea behind it is that there’s a deep assumption that there’s always a human in the loop. You’re a financial analyst and you have a laundry list of things you want to get done that day. Just like software engineers do with Codex, for every one of those items, the first thing you do is try to build it with ChatGPT, right?

Mm.

It might get you 30% of the way there or 80% of the way there, depending on the task, but there’s always this presumption of humans in the loop who are presiding over that work. That made it very easy for us to build a product where we could pass off the responsibility of auditing and review into people’s hands and responsibilities.

Mm.

That worked well for us, and it was easy to communicate, so that was wonderful. I will say we’re thinking very deeply about what comes next, and what comes next looks different from a productivity tool that’s always being monitored by a human. As we think about what comes next, there are two things we’re thinking about. One is, how do we shape the security and safety profile of the product into something that people understand intuitively?

Mm.

Something that people understand very intuitively is how to work with other people. Our thesis is that if we make the product very similar to how you work with your coworkers, your interns, and people around you, that’s something people can pick up and learn very quickly and easily. Therefore, we need to shape the security profile of the product to look like a person.

There are a few things I’ll call out there. One is the notion of read/write. Even for people, if you’re a new software engineer at a company, you often need something called readability. When I was at Google, there were checks and balances and systems in place before you could start mucking around and touching the system.

Mm.

There are already systems in place to monitor people’s work. If models behave more like people, then you can leverage those systems, which makes it very easy to talk about and work through this with financial institutions. That’s something we’re leaning into pretty heavily.

The very last thing I would say is that, at least in the context of banking, which is where we started, there’s already a system in place for junior analysts doing a lot of the groundwork. Their work gets passed off to associates, who review it. Then the associates add their secret sauce, their magic, their reasoning, and that gets passed off to the VP. The VP’s work goes up to the director and managing directors.

There’s a hierarchical system where implicit work checking is already happening. It’s been built out to try to prevent mistakes that humans make, right? Humans make just as many mistakes as models, and those systems are already in place. We’re also leaning heavily on those systems when it comes to building financial products.

Okay. I’d say there’s a misconception that, in the finance world, one mistake really compounds. Your whole career is at risk if you submit something wrong. How should people think about getting started and using it? If you’re an analyst, how should you use these tools? Do you have any analogies for where we’re at right now? There are levels of autonomy and self-driving. One of your slides, I think, said 2027.

Yeah.

You see “No more PowerPoint.”

Yeah.

A lot of people will be using AI tools in the loop.

Yeah.

How should people think about where we are now and where things are going? How should they adopt these tools if they have that misconception of “one mistake, my career is over”?

Yeah.

Obviously, there’s a human in the loop. But how should they also think about it going forward as they need to offload more and more?

Yeah, no, that’s wonderful. I have two perspectives on it.

Yes.

They start with different assumptions. I think the notion that my career is over if I make one mistake is incorrect, and I have a line of thought that follows that. But let’s first assume that it’s true. In that case, what I would say is, not all mistakes are created equally.

Yeah.

And not all model capabilities are created equally. When you think about the mistake that a model or a human can make that could be career-ending, it’s usually once you get up to a certain level. If you’re a VP, the stakes are higher than when you’re a junior analyst because there’s a presumption that the work has been checked up to that point. So if an MD sees a mistake, that’s a huge problem. A lot of the safety nets have failed for that mistake to make it up to an MD, right?

But the career-ending mistake is when you get a piece of information factually incorrect, and that factual information changes the conclusion. That’s obviously the biggest type of mistake you can make.

Yes.

We’re seeing a lot of progress in attention to detail. “Hallucination” used to be a big term that came up in 2022 and 2023, and it doesn’t come up nearly as often nowadays. We’re starting to see huge improvements in the reliability of models when it comes to factual mistakes. Models are getting much better at avoiding the kind of egregious mistake that would get you fired, and they continue to get much better.

And so I feel quite confident that a combination of that continual improvement, plus the existing system checks in place, will make the chance of that type of mistake propagating up to an MD very low.

3. Teaching Models Human Taste

What I would say the models are lacking today, and still need to improve on, is not the career-ending mistake, but the part that involves human judgment and taste. It’s, “This slide looks ugly.” It’s not wrong, but it’s formatted poorly, the colors are a little off, or something about it looks sloppy, and therefore it’s not a good slide. That’s not a career-ending mistake in the way that a factual mistake is, but it still matters in terms of how useful that work is.

As we think about the progression into 2027, I expect the model to get much better at the human-taste component. That’s what I was alluding to in my keynote: I think the model will get much better at human taste on things with softer objective functions. It’s not a question of whether it’s right or wrong; it’s comparative. Is there something better than this? I think models are going to get much better along that dimension.

How so? How do we solve taste? How do we train taste into the models?

Very early on in the RLHF days, the only thing we trained models on was essentially taste. There was reinforcement learning through human feedback, and the human feedback was always a pairwise signal: “Given 2 options, which one’s better?” The early days of reinforcement learning were built on the notion of training on human taste.

Then, obviously, we developed things like verifiable rewards and environments, and built a lot of machinery for other types of rewards and other types of RL. People kind of forgot that the early days of reinforcement learning were all about human taste.

How do you train on human taste? I think the technical answer is quite simple, but the practical answer is more complicated. You train a reward model. You get a bunch of humans to look at pairs of things and say which one is better, and this still works.

We did this for slides. How do you make these models better at generating presentations? We get a bunch of humans to look at pairs of slides and say which one looks better. Humans are really good at aesthetic judgment, actually much better than models.

Yeah.

That’s something we discovered: a human can look at 2 slides and, within seconds, say, “This one looks better.” If you have a grading model try to do that, it’ll spend a whole bunch of reasoning tokens—this pixel is this way, this thing’s that—and then maybe come to the wrong conclusion. Humans are still really valuable in making these models better. We will rely a lot on human preference to imbue things like taste.

The last thing I’ll add is that there’s precedent for this, which makes us confident this strategy will work, and that precedent is coding. Coding isn’t fully solved, but let’s say we’re 80% of the way there, and we’re making progress on the last 20%.

Coding is a combination of verifiable rewards and human taste. The verifiable rewards are: Does the code compile? Are there syntax errors? Is the output of the function expected? There are verifiable rewards in place. But a lot of good software engineering is about whether this is clean system design. Did you build a system that’s maintainable over time? Is it easy to unit test? Does it follow paradigms that the vast majority of the community or industry thinks are the right paradigms?

A lot of financial work looks very similar from that perspective. You have an Excel model. Does it sum up? Are there reference errors? There are a whole bunch of verifiable things we can check to make sure the work passes a first-pass sanity check. But there’s also the question, “Is this the way I would have built this model?”

Something I learned about as I started working in finance is that there’s this notion of the feel of the model. You perturb the inputs and look at the outputs. An experienced financial professional can perturb the inputs, look at the outputs, and understand whether the data flows through the model in a way that feels good. That’s a human preference that looks a lot like asking whether code looks clean and is well-written. I think our success in coding gives me confidence that we can solve a lot of these problems in finance as well.

Yeah.

Yeah. We can shout out Frontier Code from Cognition. SWE-bench is very much, “Is the answer correct?” This is also, “Would you merge the code?” You can have multiple answers that are proper PRs and pass.

Right.

But which one would you merge? That’s why you sometimes see the charts where lower reasoning is better, but it’s a taste benchmark as well, right?

Yeah. One of your other 2027 predictions I thought was interesting was that the models you’re training will become much better at investment theses and generating ideas for investment and trading. But obviously, that’s something humans, or experienced financial professionals, are very good at, and they have an intuition for it. It’s often counter to the prevailing wisdom out there. Otherwise, there’s less chance of it making a great return. How are you tackling that problem in terms of training the agents to do that kind of work?

4. From Analyst Tasks To Investment Ideas

Up until today, the vast majority of applications of AI in finance have been really focused on the junior analyst’s work. I think that’s for 2 reasons. One is that you have to build the product based on the capabilities of the model today. If you look at those capabilities, they’re really quite good at taking pretty concrete instructions and executing. “Build me a slide. I want it to look this way. Make it look good. Use our template.” The model is quite capable of doing those things, so we’ve built product experiences around them.

The other thing I’d call out is that there’s a strong preference to make the model good at things humans are not good at or don’t like to do. Most people I talk to really don’t like building Excel models. It’s tedious, detail-oriented work, and it’s easy to get wrong. Similarly with slides: you’re copying logos and placing them very precisely on a specific part of the page. That’s tedious work that a lot of people don’t like to do.

That’s been part of the impetus for us to focus on some of these tasks. It’s a combination of 2 things: the existing model is pretty good at certain types of tasks, and human preferences about what people like to do and don’t like to do help drive the direction of the product. It’s a combination of those 2 things.

I think we’ve solved a lot of the low-hanging fruit in those areas. Going into 2027, we’re going to start to move up the stack: from tasks a junior analyst can do, to tasks an associate can do, to tasks an MD would do. As you move up that chain, the objectives become much fuzzier. It goes from, “This is clearly a good or bad slide,” or, “This is something I would put in front of a client,” to, “I don’t know if this investment idea is good.” It would take 3 years of waiting to really give you a clear, concrete answer about what the market has done.

The objectives start to become fuzzier, and the way you train the model and think about making it better at those things becomes more difficult. It becomes more difficult from a research perspective, but it’s not impossible. It takes a lot of research focus, wonderful data, and feedback from experts in the field. We really want to get the P99.9-type feedback to make the model as good as humans. I don’t think it’s an insurmountable problem. What it takes is continued scaling of the base model and focus.

Focus on the research labs to think that this is an important problem to go solve. One of the things that I’m doing on my team is making this an area of focus and priority for OpenAI. Because of the combination of those 2 things, I feel confident we’ll make a lot of strides in 2027.

Do you have any comments on the current financial benchmarks? I think we’re still at the stage of asking, “Does this check out numbers-wise?” How do you think about making your own evals? I think this is also an open call for people working in this space to work on their own benchmarks.

As a follow-up, you said to build systems for models today, but you also make a lot of predictions for the future. How do you put guardrails, limitations, and checks on the financial services you build for people today, just so they’re confident it’s not going off the rails?

5. Financial Evals Need Better Measures

Yeah, makes sense. Let’s talk a little about evals. Evals are something near and dear to my heart. I spent a lot of time working on them. The prior I would share with people is that building a good eval is incredibly hard. That’s not obvious to people who haven’t worked on an eval before. Building an eval that actually correlates with the thing you’re trying to measure—the skill or capability you’re trying to measure—is very difficult, because whatever you put together is just a proxy for that thing, and it’s just one dimension.

Second, it’s very tedious to get the answer key or the way you grade things right. If you don’t get the grading right, it’s an eval that’s very easy to hack, a very easy reward hack. You could score very highly on the eval but actually do really poorly on the underlying skill or objective you’re trying to measure. The prior I’ll share is that evals are really hard, not just in financial work but across the board.

We don’t think there’s a really good eval today for financial work. There’s a bunch of existing prior work. I’ll reference some of the work that we did, which was GDPval, a little more broadly in knowledge work. If you guys look at the Astra announcement, we don’t talk about GDPval. We know the eval is completely saturated, and the pieces that aren’t saturated are essentially broken, so we don’t report on it. We don’t track it anymore.

One of the tricky things for GPT-6 Astra was that we didn’t talk a whole lot about financial evals. We have some internally that we didn’t talk too much about or want to share at the time, but they’re coming. We didn’t have a good way to show progress on financial tasks with these evals. There’s a funny bias in evals: if the numbers look good, researchers assume the model’s gotten better and don’t investigate it. If you’ve made a much more capable model and the eval numbers look bad or haven’t increased as much as you want, then you go investigate.

Something like this happened with GPT-5.2. We ran it on a set of financial evals, and the numbers didn’t really change much. We thought, “What’s going on here? Let’s take a look at this eval.” We did a deep dive into the different evals, and lots of things were broken. Basically, we found that we had saturated the evals, and the questions we were getting wrong were driven by incorrect answer keys, poorly specified prompts, or some other flaw with the eval.

One of the things my team is currently working on is trying to build a much better eval that actually predicts the ability to automate the work of a specific profession. The idea is to have something like an investment banking eval that predicts that, if you score well on it, you could probably automate a lot of the work of a junior analyst in investment banking.

We’ve got to wrap here soon. Any call-outs for people in the finance space in the Latent Space audience? Your team is probably hiring, but the last thing to put out there is: how can people be helpful to you?

Wonderful. First and foremost, please check out ChatGPT for Financial Services. It’s an enterprise product, but we’re trying very hard to make it available to more prosumers on the Pro plan, so hopefully we’ll have some cool announcements there.

Second, yes, my team is hiring. We’re seeing incredible customer traction with the product, and because of that, we’re hiring and scaling the team rapidly. It’s a very diverse team with lots of different roles, so I want to call out a few of them very quickly. There’s the standard front-end and back-end engineering—traditional product building.

Second, we’re very much looking for wonderful product managers. In some ways, great product management is the main bottleneck in trying to win in this space, so we’re trying to hire wonderful product managers in investment banking, equity research, asset management, and across the board.

The other role I want to give a special shout-out to is a subject matter expert. One of the things we did well in coding is that the people training the models and making them better understood what good code looks like. That was a huge tailwind for us as we made the models better at these things.

We don’t have that advantage when it comes to making the models better at finance, so we’re building out and trying to hire a lot of folks who can bring financial expertise and work with the engineers, researchers, and product managers to make the model and product better in their specific field. If any of those roles resonate with you, I highly recommend that you reach out to me. I’m available on LinkedIn. We also have these job listings posted on our careers page. They’re pretty easy to find.

Great. Well, you answered my question. I was going to ask when I can have a play with this, but it’s coming to prosumers soon.

Yeah, we’re working very hard to make it available to more people.

That’s very common OpenAI stuff. It makes sense: you make stuff for enterprise, and then it trickles down to the regular models. It comes to the Pro plan, so expect that for your general usage, it’ll get better at financial stuff down the line.

Yes, absolutely.

Awesome. Well, very excited to have you. Check out the keynote. You gave a lot of good predictions.

Yeah.

You talked a little bit about some juicy “we fixed pre-training” issues, so I’ll leave breadcrumbs for you guys to go watch. But thanks for joining.

Yeah, awesome. Thank you so much.

Awesome. Thanks.