[State of RL/Reasoning] IMO/IOI Gold, OpenAI o3/GPT-5, and Cursor Composer — Ashvin Nair, Cursor

Okay, we're here at NeurIPS. We're recording a special Latent Space episode from NeurIPS, and we're here with Ashvin from Cursor. Welcome.

Hi. Yeah, thanks for having me.

So I guess Cursor is a new identity. I didn't even know if I should say that, because you only joined Cursor 3 months ago. Before that, you were at OpenAI, where you worked on o3; before that, you did a Berkeley PhD in RL, but focused on robotics.

Robotics. Yeah.

Is it weird switching from robotics to language models?

Okay, this is kind of interesting because a lot of people have been doing this. I mean, OpenAI—yeah, robotics. I was actually at OpenAI in 2017, also working on robotics. I was interning right before my PhD, where I worked on robotics there.

2017? He was famously OpenAI's first intern.

Oh, really? Okay, then he might have been before me. But yeah, there were 15 interns. It was a very different company. It was just robotics, Dota, and 15 interns that summer, all having pretty exciting individual projects. That set of interns—if you look at where they are now, it's kind of cool.

Yeah. Was there anyone from that class that you would shout out?

There were just a lot of cool papers that came out. Lerrel Pinto is now at NYU.

Yeah.

The person who leads reasoning at xAI—I forgot his name.

Well, he left. Eric?

But yeah, I forget his name. He worked on K-FAC and stuff, I think.

The vision dude, Greg.

Not Greg, but yeah. It was an exciting time to be there. I think robotics is a pretty good fit for LLMs because the switch ends up being pretty similar. You want to look at a lot of data, and it's hard to get stuff working in the robotics world. I think it builds very gritty people who look at data a lot, that kind of thing.

So, for whatever reason, I think that transfers. It's happening a lot, and I think it makes a lot of sense.

One of my NeurIPS highlights so far was having dinner yesterday with Lex Fridman. It was a small group dinner, and Lex used to be in robotics. He gave his assessment of robotics people: robotics people are the best to talk to at NeurIPS because they're the most well-rounded.

Because they don't have a choice. They work with the real world and real-world-looking data. The most unhinged, the most detached from reality, are the simulation people.

I see. Yeah, yeah, yeah, yeah.

I think I agree. I actually did a little bit of both during my PhD. I worked on prototyping ideas in simulation and then getting them working on real-world robotics. Robotics is probably where you feel AGI the least, because it's just so far away from working.

Over the last year, there have been demos that have been super interesting from Physical Intelligence and Sunday and stuff. I'm starting to think, “Okay, this kind of stuff—”

Have you seen the Sunday robots themselves?

I haven't seen them. Apparently, they've been doing demos, and I'm pretty keen to see them.

Yeah, I've seen the Physical Intelligence ones live, and it's pretty impressive. Just in someone's living room, folding laundry and stuff—you can just toss it in there. Everybody must be mesmerized.

Yeah.

Okay, one last thing on robotics, and then you can pivot to OpenAI. OpenAI is restarting a robotics team. Is that serious?

I actually know very little about it, because I was in a pretty different part of the org.

Yeah, I mean, I think it's serious. I think there's a ton of excitement around robotics right now. I'm actually curious what drives it, because I don't think I fully understand. There have been crazy raises recently for robotics companies, right?

I guess my own view on it is that when I left robotics in 2022, I thought I would actually come back to robotics. But my view on it now is that LLM agents are going to be a trillion-dollar market before robotics is maybe even a $10 billion market.

This is because LLM agents already create value out in the world. With robotics, it's hard to make the case that AI robotics does anything that useful yet. Once it does something useful, you have to make the unit economics work out, and I think that's also quite hard. Reliability, fixing these robots, and those kinds of things—it all makes it difficult.

I would say the market is kind of efficient in that software LLM companies are raising tens of billions, and robotics companies are raising hundreds of millions.

I think very recently it's been single-digit billions.

Oh, really?

Yeah. I think that's the surprising thing to me: it feels ahead of where it's actually at.

I would say robotics is in kind of the GPT-1 to GPT-2 era right now. I haven't worked on robotics. What task would qualify as, “Oh, that's the inflection point”?

It's a little bit like—you know it when you see it. I thought the Sunday demos were kind of cool. Maybe it's starting to get there, where the details matter a lot. It can't just be in a new scenario, one that you haven't seen before, and maybe on—

Generalization?

Yeah, exactly. I think that was kind of what GPT-2 was too, right? You start to see hints of cool generalization. It doesn't have to work out of the box, but at this point, especially, it still feels like in robotics you're not exactly investing in a technology; probably you're just investing in a team.

Yeah, yeah. I'm not in the space whatsoever, but that's kind of my impression.

It's actually nice when you're not in there, because you're as informed as basically everyone else. So we just kind of speculate.

Exactly. There's a robotics team at OpenAI.

So, coming back to language models, did you join OpenAI, or were you—

I joined right before ChatGPT, in September 2022.

Yeah.

I was pretty burnt out from my PhD, and I thought, “Okay, I'm going to go to this chill research lab.” Then ChatGPT happened, everything blew up, and a lot of stuff got refocused.

What did they tell you they were looking for you to do? Obviously, ChatGPT surprised OpenAI, but—

I joined on the CodeGen team.

The Codex team?

Exactly. It was the team that shipped Codex, but by the time I joined, we were more so working on the model, doing tool use and those kinds of things. It was very related to ChatGPT; we were kind of a sister team to the team that made ChatGPT.

We were working on making the models smarter through programming competitions, how to do SFT for that, and that kind of stuff.

And IMO gold has felt reachable in that time?

Oh yeah, crazy. If you told me that we could have gotten IMO gold, I would have assumed that we could all just go on vacation. It's all over—AI is solved. There's no point in working anymore.

We got it.

Yeah.

It feels like nothing's changed that much. Life is still the same.

Yeah. I think that's super interesting. I don't have a great way to explain it, but that's actually what I spend a lot of time thinking about: why is that the case? You see this again and again in AI, with solving chess, and then it doesn't really matter, solving Go, and—

Yeah, so you keep seeing it, but it surprises you every single time. I think, first, we keep moving the goalpost.

Yeah, we're very good at that.

And second, I think our definitions of what constitutes AGI are bad. We don't actually mean what we say when we say, “When we have achieved this, then we have AGI.” Clearly, when we've achieved IMO gold with a language model, we have AGI. That's wrong.

Yeah, and I think shifting the goalpost to some extent is correct. We keep Goodharting whatever goalpost we have—

And I think it's kind of hard to—

To say “Goodharting” is too negative. It's like, “I will cheat to do what you asked me to.”

Mhm.

But I don't think it was cheating.

It was just scaling test-time compute at a meta level. I think the community is not cheating, but it makes a lot of implicit decisions to go after the eval benchmarks that matter the most. And so, yeah, for sure.

Yeah, exactly.

But yeah, hopefully I’m not that Goodharted.

Well, it kind of clearly is to some extent, right? Most programmers in the world cannot do AI at any decent level, but we’re still struggling to automate most programming jobs. There’s a lot of stuff left to do, so language models are here at the junior-to-senior developer level, and then suddenly for AI, you’re like—

Exactly, and there’s something suspicious about that.

Yeah. Okay.

I kind of saw this at a meta level with RL research also. I did my PhD with Sergey Levine at Berkeley from 2017 to 2022, and that era of RL research was super interesting because it was super hyped, starting from DQN in 2015. A lot of the methods that people were really excited about were off-policy learning, value functions, and these kinds of things.

Somehow, that stuff hasn’t really panned out, I would say. It’s not exactly clear why, but in the academic literature, we thought we were making a ton of progress. In retrospect, I have to say that we probably overfit to the benchmarks pretty heavily.

The way I see this in retrospect is that we gave ourselves a lot of new knobs to tune and then implicitly tuned those to fit the benchmarks. Everyone knew at some level that we were doing that, but I think it’s hard to appreciate that it’s not just happening for a single paper at a meta level—it’s happening for the whole community, too.

Yeah. And I think the result is that a lot of the RL research that came out of that era isn’t used that much. I think it’s for a similar reason: basically, we were doing benchmark maximization.

I will flat-out say there was an RL winter. Entire startups were founded based on the premise at the time and basically gave up.

Mhm. Some of them died, some of them pivoted, whatever.

Yeah, yeah, yeah. I think because I was in academia, there was still quite a lot of excitement over it. But it still felt quite academic, and I was a little bit frustrated in that era because I felt like one of the pitfalls of academia is that it doesn’t really reward simple ideas that work. Instead, it tends to reward math-y ideas.

Those math-y ideas also give you implicit knobs to tune that allow you to overfit, while the things that actually work tend to be simple ones that have fewer knobs and just generalize to many things.

There’s just less secret sauce to it apart from throwing a lot of compute.

Exactly. Exactly. But those are things that tend to—

It’s not intellectually interesting.

Yeah, exactly. From an academic point of view, it’s like, “Why am I sitting in school?”

Yeah, yeah. I think for a lot of people who do PhDs, they’re wired in a way where they want to think about interesting new stuff.

And yeah, the scaling era kind of speaks to that.

Scaling era. Is the scaling era over since we’re up?

Well, I think I’ve just been pulled into that from an interview. I don’t think it’s over, but there’s definitely something interesting happening, right? The thing I was saying about AIME and IMO—I think we’ll still continue more or less on the same track.

Clearly, these labs are releasing their new pretrained models, and they’re still doing much better than before. So I think scaling is still happening, but I think it’s happening in a different way.

It’s worth seriously interrogating why we’re not just automating all jobs right now.

I think my view is that RL, the way it’s applied to LLMs right now, is kind of a weird, funny tool where it doesn’t really generalize beyond the training distribution that much. It generalizes to some extent, and it generalizes in interesting ways, but it’s very peaky, right?

It can kind of kill the training distribution completely. It can be the best in the world at it with not that much effort, really, but it doesn’t really generalize. So I think what we have to do is bring the world of economically useful tasks into the distribution for RL.

If we commit to using RL as a tool—and it might be the case that maybe there’s some cool continual-learning thing or something that shifts the paradigm next year—

But—

It really feels like if RL is a tool, then a big thing that needs to happen is that the intelligence of the models isn’t the bottleneck. It’s more that you need to have products that bring the entire context of what someone wants to do into the product so that the LLM can see it, and then you need to do RL on top of that.

Yeah.

Have you seen GDPval?

Yeah, I’ve seen it. Yeah, yeah.

Is that basically what you’re envisioning?

Yeah, I haven’t looked at GDPval closely. Actually, I haven’t seen exactly—roughly, to recap, it’s 128 tasks across white-collar jobs that account for more than 5% of GDP, right? They basically created all the context for the eval and evaluated every model.

Famously, OpenAI’s evals—whoever runs that one—always finds that Anthropic’s are the best for—

Yeah, yeah. It’s, uh—yeah, props to them for publishing it.

Doing that. It’s actual science.

I think it’s good.

But in a sense, generalizing beyond coding competitions to economically useful tasks—

That is it. I think that is what’s more important for GPT-6.

Yeah. What I’d like to do is—I just haven’t read the GDPval traces closely. It’s not clear to me what the job of an accountant entails and what kind of context needs to be in the product so that you can do it.

PDFs. I see. I see.

They try to go as close to source documents as possible.

I see. I see. Yeah, yeah.

So yeah, I think roughly operating in this kind of thing is what I envision, because it can’t be an artificial thing like, “Oh, let me clean up this data for you to make it easy for the LLM to process.” No.

PDFs in an agent loop. Yeah, I think that’s roughly the right shape of the thing.

I guess how I imagine this being operationalized is that you’d want to co-design the product and the model, so that the product—whatever it is—can provide the right context.

Coding is maybe the easiest first step, because most of the context that you care about is just your codebase, being able to run stuff in the terminal, and that kind of thing. And still, we’re not that close to automating it necessarily.

But for all the other jobs, the context is insane, right? It’s all the conversations you’ve had with your coworkers, your Slack messages. For my work at OpenAI, I was working on hyperparameter-scaling research, and I actually wrote not that much code.

Grid search or neural architecture search?

No, more like understanding the scaling laws of deep learning in 2020, where it was, “Oh, you have to initialize the layers in a particular way to get good scaling”—kind of the analog for that for RL.

Okay.

The thing is, I didn’t write a ton of code. Writing code wasn’t the bottleneck. It was more that, over the course of a year, I would run sweeps, look at the interactions between different hyperparameters, and build up that knowledge through a year of looking at different graphs.

To do my job, the model would also need all those things in context to successfully automate my job. You would want a product that allows you to bring all that context in.

Did you have to build it for yourself, or is there an existing one?

No. Those graphs were just sitting in my head.

Yeah.

Right. So I think it would be pretty hard to automate that job. What you need to do is build a product that brings that context in, and then you want to do RL on top of that to teach the models to use that context.

Yeah. Another conversation that has really come to a head this year is the death of “one model fits all.”

I feel like the point of the G in AGI is one model fits all.

Mhm.

I think OpenAI has clearly abandoned that this year.

Oh, what do you mean?

Fidji wrote a blog post titled “We Are No Longer Doing One Model Fits All.”

Okay, interesting. Okay.

And I think Mark Chen, or one of the other senior people who aren’t Sam, also said this in a podcast. Basically, the idea was that you started with Codex, someone else was doing InstructGPT, and then we launched GPT-4—

I guess o1.

And o1 was kind of supposed to be a reasoning, one-model-fits-all system—

And we merged the GPT-4o and o1/o3 lines into GPT-5—

And now we’re splitting it out into GPT-5 and GPT-5-Codex again. It’s just a weird—

Well, OpenAI is very guilty of that. I don’t think you should interpret those as scientific facts about the universe.

It’s just more like OpenAI has a tendency to shift the org chart, basically.

Yeah. Right. The world has a tendency.

Yeah, exactly. So I think a lot of it is related to that. But yeah, I see what you mean by the current reasoning paradigm fitting itself to this kind of peaky-in-certain-areas thing, right? I don’t think it’s so much a matter of model capacity, though.

It’s just more of an organizational thing: if you care a lot about coding, you probably don’t have the data to do all the other stuff. I don’t think it’s so much a matter of—if you had all the data, you would probably benefit from just training on all of it, and you’d get some generalization between these—

But it’s hard to find one organization that cares about all of these at once.

Yeah. Yeah.

Yeah. So before I double-click on the o-series in OpenAI, I do like to ask OpenAI people who were there: do you have a favorite blip story?

Yeah, the blip was crazy for me. I was at Thanksgiving—

Everyone remembers where they were and what they were working on.

Yeah, exactly. I was at Thanksgiving with 2 OpenAI friends, actually, and one of them, on Friday afternoon, was like, “Oh, Sam Altman just got fired.”

We were just working together, so I was like, “What? Oh, haha, good joke.” And then, yeah, it was crazy. It was just a crazy weekend of ups and downs. We thought—

You signed a letter?

Yeah, I did. About 95% of people signed it.

Yeah, yeah, yeah. I thought you would move on to Microsoft, or—

Well, I think maybe I had a slightly more complicated reaction. I actually do think that governance feels really important to me, because it does feel like, no matter if we hit AGI in 2 years or 10 or whatever, it’s not clear that we have a good structure for the governance of it.

Okay.

And so it is a question that we probably should spend more time on. During that period, I was pretty willing to be like, “You know what? Let’s forget about the equity and stuff. I think it’s good and healthy to have a conversation about how exactly the governance should work.”

Okay. You care about this.

Uh-huh.

Right. So now the OpenAI nonprofit has this secret shadow board of members that determines when we’ve reached AGI.

Yeah, yeah.

Is that better?

Yeah, I don’t have an answer. It’s just—it’s not fair—

Above my pay grade, but—

Yeah, and even back then I was kind of like, well, I don’t care. I—

I do care quite a lot. When the blip happened, one of my reactions was, well, you know, this nonprofit board stuff—if it takes somewhat surprising, maybe erratic actions, maybe you’d rather just have a thing like the Microsoft board, which is probably all the pensions of the world. Why? Serious people, but also the stakeholders are kind of the whole world, because everyone is, through their pensions or something, invested in it.

Maybe that is a bit more of a democratic way to run things than having 7 people run it. But yeah, I don’t really know. It feels like we haven’t solved governance at all, though, right? Forget AI—even stuff like unhealthy food or social media. It kind of feels like whatever the capitalistic incentive is doesn’t actually capture good outcomes for society, maybe.

So about the transition into reasoning: you shocked me by mentioning that the reasoning team is 300 people.

It’s kind of like, now that o3 was shipped as a product, I think it just gets larger and larger, how many people work on it. I’ve lost track of the numbers, but a lot of people contribute to the different aspects of safety and evals.

Yeah. The original o1—I saw the video, and it was like a dozen people.

Yeah, well, even then, if you look at all the contributors, it was probably more like 50 to 200 people.

Okay. So let’s tell that story from your point of view: figuring out what RL means there. And I guess, was this a branch of any prior work that you want to credit?

Yeah. Setting the scene, I guess, in 2023 people were talking about whether scaling laws were dead, that kind of stuff.

Every year, every year.

Yeah, yeah. But especially that year, it felt pretty serious.

In general, OpenAI is really good about having conviction in something and just, from first principles, going after it. I think the people who are probably most responsible for that are Ilya Sutskever and Jakub Pachocki. I think Dota 2 was more or less the same template in some ways, right? That was in 2017, and a lot of the people there have this AGI-in-their-bones kind of point of view.

They were basically convinced that RL would be the way to get there. For a long time, people had been convinced that something like that should work, and it’s just that it started to work once the pretraining got good enough.

Okay.

Yeah. I think human feedback is a bit of a side branch, because you can’t really pour that much compute into it, right? You take the model and elicit it to be a little bit better in terms of personality.

But the people there were really convinced that, at some point, it’s not about copying the internet. You can do RL, and that’s the path to getting much better intelligence. So there’s kind of a long line of returning to RL in different ways, and then around 2023 is when it started really clicking.

It was interesting because the initial models didn’t perform way better than the existing models, because they were smaller scale. But people were very good at saying, “This is kind of interesting. The reasoning trace that you see here is not something that you’ve really seen be so accurate in other models.”

It’s similar to how a lot of people didn’t really think of GPT or GPT-2 as something that was super compelling. I personally didn’t like GPT-2 that much. I was like, “Okay, whatever,” and then GPT-3 happened, and I was like, “Oh, wow. I feel a lot of FOMO sitting in my PhD.”

It’s kind of that. I think it takes a bit of first-principles conviction to decide that there’s something here and that we should really scale it up. OpenAI is really good about, once you decide that something is good, scaling it up all the way.

Was there an internal prototype before o1 that was like, “Okay, this is the thing. We’ll fund it to scale it up”? There usually is.

Yeah, exactly. It was running RL on even a pretty small model, producing very interesting reasoning traces and getting surprisingly good scores on math in a way that we couldn’t have done without a bunch more pretraining.

Once that looked good, more and more resources went to scaling up that new line, as well as adding things like tool use.

I think a lot of people make a lot of headlines about the large models, but I think the minis are very underappreciated—how well this solution works. Any comments or discoveries there?

Yeah, nothing much to say there. I was also not super involved in the mini stuff. Maybe one thing, not exactly related to that, is that externally people are like, “Research seems to come in these big leaps.”

Okay.

But internally at OpenAI, it feels very smooth. You have a bunch of experiments. Some of them have inconclusive results, but maybe you stack them.

Yeah, exactly. You stack them and just keep scaling. You keep having different runs that get a little better each time.

Okay.

So I think that’s maybe one other aspect that’s a little underappreciated.

I don’t know. In the media, there are just these wild swings between, “Oh, we’re—”

Yeah, exactly. Internally at the big labs, it’s just kind of like, “We’re chugging along.” Maybe this month is a little better than last month or something, but it’s not as crazy up and down.

I think the question is that there used to be more of this, and now I know there’s less. The stuff we’ve released—we’re internally about 6 months ahead. Part of the reason why people at OpenAI weren’t that excited about ChatGPT’s launch was because they already had GPT-4. They were like, “Oh, we just put this out. We’re already way ahead.”

I think now people are just releasing things as they have them, like—

I think, yeah, especially because there’s some competitive pressure, right?

I think people are probably pretty worried that if you let a lead linger for too long, that’ll grab a lot of market share. I don’t know—Nano Banana Pro right now is probably—it’s like a month…

So I would say the lead time from internal to external is about 1–2 months.

Yeah, which is exactly—

Tiny. Pretty short.

Yeah. Tiny. Anything else on the reasoning side? I guess you can talk about the work on coding. Was there anything that surprised you, or is there an external misconception about the o1/o3 side before we go to Cursor?

Not really. I think it felt by early 2024 like, “Oh, wow, this recipe really works, and we can see how far we can take it.” It was very steady progress, and by that point it was probably pretty predictable that we could really smash things like the IMO or IOI.

One funny thing that happened while this was going on is that I went to a conference called The Curve.

Yeah.

It was about AI progress—

And Joseph Gordon-Levitt—

Yeah. I went last year. This was before the o1 stuff was released.

Yeah.

I went to this thing where people were making bets on where we would be on Epoch AI’s FrontierMath exam and Humanity’s Last Exam and things like that. Their estimates were, “Oh, we’ll be at 10–20% in 2027.” At the time, there were already models internally that were better than their estimates, so they were off by about 2 years or something.

The interesting thing is that these are also people who were predicting that there would be Dyson spheres by 2035 or something.

Okay, so you see what I’m saying? Their current estimate is way under. Are you too pessimistic in the short term and too optimistic in the long term?

I don’t know. There might be Dyson spheres by 2035. I don’t regard it as impossible. But I think that is one interesting aspect: people still seem pretty miscalibrated in different ways. I do really appreciate how that community makes predictions, because most of the rest of the world just cynically says, “I saw this the whole time.” I appreciate that they actually make predictions.

Is this EA-adjacent?

Yeah, exactly. It’s that group.

Yeah. I like that they register their opinions ahead of time.

I think broadly, the people who’ve made capabilities predictions in that group have been broadly correct if you look from 2015 to 2020 or something. A lot of people thought AI was a sham, or that it wasn’t really going to be useful for a long time, and actually it is. It’s somewhere in the 2030-ish range that it will probably reach human-level intelligence.

Yeah, it’s weird. I feel like a skeptic when I keep saying that everyone always predicts AGI happens in their lifetime. That’s very convenient for whoever is making the prediction, and we have a consistent view of history where you can see people in the 1800s and 1900s making predictions. It somehow always lands in their lifetime, whatever the thing is.

But this time it might happen—almost surely, right? I don’t know.

I’m pretty sure here.

Yeah. It’s an interesting observation: how different are we from our predecessors in terms of developing a technology? Did the DeepSeek moment earlier this year—which was also crazy—change anything internally?

Not really. I think what surprised me more was that it created such a moment. It was kind of confusing, right? DeepSeek shows that NVIDIA chips are actually more useful than previously thought, and NVIDIA’s stock goes down a bunch. It was kind of—

I think the steelman of that side is that you don’t need the top-of-the-line NVIDIA chips. You can use the previous generation, or the restricted ones they sell to China, to do an equivalent amount of work for a reasoning model.

I see. Yeah. But I think the feeling at OpenAI was that we had a better model already at the time, right? Smarter models were clearly quite valuable, so you wanted to be at the frontier.

Okay. I wasn’t quite framing this as a race-dynamics thing between labs. It was more, “Were they right? Were their approaches right?”

They had R1, which was a really cool branch. So it’s more commentary on what we learned about RL this year.

Yeah.

It does seem like a lot of the labs have converged on some similar way of doing RL, and they’re all kind of back at the same level of being at the frontier again. Even the Anthropic models, like Opus 4.5, have this kind of ARC-AGI-2 plot that looks exactly like the OpenAI ones, right?

I think everyone seems to be converging on a pretty similar form of RL. It’s interesting. People basically figured out, in one way or another, how to achieve more or less the same thing.

Yeah. Let’s talk about the move to Cursor.

Yeah.

Why is Cursor accumulating and drawing in so many cool RL people?

From Cursor’s perspective, it’s nice not to be so dependent on external labs for everything. I think there are also unique opportunities to co-design the product with the model in ways that we couldn’t unless we actually built the model ourselves and had access to making it good.

Yeah.

That’s broadly why Cursor is so excited.

Okay, I’ll push back a little bit. OpenAI has infinite resources.

Infinite data, and it has Codex. You could have just stayed.

Yeah. Yeah. Well, actually, right around when I was leaving is when people started really using Codex a lot. That happened right after I left, so that was kind of funny. Mostly, people were using Cursor internally—

Maybe a bit of Windsurf because it was left over from the previous thing.

Sure. Yeah, exactly. So it wasn’t that obvious. But more to the point, RL is kind of a tool that doesn’t generalize that well. What you want to do is bring the entire test distribution inside your training distribution. I saw the opportunity to do that directly at Cursor.

I think the Cursor folks are also really excited about that vision. It’s a small place where the product people sit right next to the ML people, and I think there’s a lot of potential there.

You can see that recently Jacob Jackson had a blog post about online Tab, where we’re doing policy updates—

Which is every 2 hours.

Exactly. A policy update every 2 hours or something. That’s the type of thing that’s a little hard to do. It’s very hard to imagine that at OpenAI, for example, because the product is this complicated thing, and the product people and RL people are on different sides of the organization.

I think if you put your mind to it, you would. Tab is an autocomplete. It’s a smaller model, and it’s not as complex, I guess, as—

Yeah, but I don’t think that’s really why Cursor was able to do it. It’s more about the organization itself being smaller and a bit more focused.

Yeah. Since you’re indulging this, I think the question about continual learning—which has always been a big theme and is an even bigger theme this year—is: don’t you need to curate your data? You can’t just chuck whatever your users are doing straight in, because that tends to get you toward the middle of the distribution. You actually want to spike it.

I guess it depends how you’re thinking about continual learning. Humans are quite good at dealing with bad data too, right? You can see someone doing something dumb and decide that you’re not going to do it.

Filter it out.

Yeah, but it’s not even actually filtered out. You presumably have some kind of value function that says, if you see someone touch a hot stove, you’re not going to do it. It’s not just filtering it out; you’re actually not going to do it.

You could rediscover hot stoves from first principles.

Yeah, but you don’t need to. I think there’s something pretty deep there. It seems like we’re a few orders of magnitude of data efficiency away from that kind of thing. You do something once, or you make a mistake—if you introduce a bug in your code, you’re not going to do it again. But the models will happily keep doing it, even within the same context, and definitely across contexts.

Yeah. I think there’s something interesting and deep there. I suspect that it’ll be paradigm-shifting in the next year or something, but I have no idea what it might be.

Yeah.

So primarily, you’ve worked on Composer, Tab, and maybe Search?

So I’ve actually just worked on Composer, and that’s kind of the main focus of the company, basically—or at least the ML group—is shipping a better Composer.

Can you describe, I guess, the impressive brag a bit about the ML group?

Yeah. I think the ML group is great. It’s just 20 or 25 people, and I was honestly very pleasantly surprised at how good Composer is, given the size of the group. It’s not like a big research lab yet, and I think it’s a really good model.

You can see that in the reception, and I think it’s the start of hints of co-design with the product in some ways. One of the reasons people really like it is that it’s smart enough that you actually want to use it. It’s also fast, so you stay in the loop with the model while you use it, because all the other smart models are slow enough that you want to context-switch away and come back. That sucks. As a programmer, it just sucks to context-switch; it kind of gives you ADHD. It’s really terrible.

I agree. It’s one step in the direction of being able to be more in sync. I think the whole company is full of people who want to code. Even the co-founders actually code, and the co-founders are often some of the best high-taste testers, which also gives you a lot of reassurance that you’re going to ship good stuff.

Is there any example of a task that Composer doesn’t solve yet but that you’re really motivated to solve?

Ironically, I feel like I’m actually a low-taste tester in some ways, because I just write slow machine-learning code and think about algorithms all day.

Yeah.

I think more broadly, I’m super excited about co-designing the product so that you can actually—not just right now, where we’re getting better and better at answering user prompts. I think that’s why Composer 1 is quite good.

What we’re really aiming for is to automate software engineering as a process, where you write code, go look at Datadog, look at what’s happening, then come back and maybe have some hypothesis about what’s better and rerun things. That’s the type of thing that we actually want to make the model do.

I do think that Cursor is uniquely positioned to do that. If a lot of what a software engineer does ends up in the product, I think we can use that to get better and better at not just writing code, but the whole job.

I think that’s very inspiring. Just to double-click on any sort of RL insights: Sasha and Lee have talked a lot about the internal tooling that you’ve had for all the cluster visualizations. Is that helpful? Is that what every lab has?

Yeah, I think the tooling at Cursor is actually really good. People are just down to vibe-code stuff and test their own stuff, so we have a lot of good tooling where you can have an SSH session into our own user environment or something and see whether code runs the way that users got it to run. I think that’s quite nice.

One of the big lessons in ML in general is that you want to be really close to your data and understand your data well. I think we’re again uniquely positioned to do that well, especially because all the internal tooling is just internal—you’re not buying anything.

Yeah, it’s just internal. Part of it is that we’re also working on a product where you can understand it really well because it’s a code product. If I were to look at a biology question at OpenAI, I’d have no idea what it was about.

Yeah. Interesting. I think that’s a good overview of everything. Other than OpenAI and Cursor, is there interesting RL work that other people are doing that you’re still mulling over, that’s influential to your thinking? Good papers, anything like that?

Unfortunately, I’ve gotten in the habit, especially at OpenAI, of not reading that much external work and just reading people’s internal Slack posts as the main way to learn new stuff. No super-inspiring recent things have popped out to me.

I do think that this vibe of continual learning feels like there’s something super interesting there. It feels like maybe even people in academia could make a big crack at it.

And continual learning specifically means kind of what Tab is doing?

Maybe what Tab is doing, but also just in-context learning with infinite memory or something. Once you experience something in context, it should just be in your weights, and you shouldn’t have to make that same mistake again. That kind of thing.

Why do you think there’s—okay, but it should be in your weights, and there’s a finite capacity for the weights to remember things. You will forget things if you do that too much, right?

Not necessarily. You start out by memorizing or learning from trillions of tokens. Now you’re going to experience thousands or maybe millions of tokens.

You only need one epoch.

Yeah, exactly. So crazy. It feels like if you could learn enough about those million tokens that you’re actually deploying on, I don’t think you should need to worry about overloading the capacity of your model. I don’t think there’s a risk of that, right?

Because you can train on a trillion tokens and it’s fine, right?

Right. So proportionately, it’s a drop in the water bucket unless you run it for years and, at some point, it starts—

Maybe.

Yeah. So basically, I find it very curious.

I’ve only had one podcast on the information theory of language models. What is the theoretical capacity? How much are we using? You should probably track that.

Yeah, that’s a good idea. Treat the weights as a hard drive. If you want to store things in the weights, treat them as a hard drive. What’s the capacity of the hard drive? How much can be stored in there? We know the capacity—it’s the number of bits represented by the parameters.

Yeah. You physically cannot store more than that.

Yeah. I’ve heard that someone recently at Cursor, Jacob, brought up this view. I don’t know if it’s a more public view, but there’s kind of a hard-drive view of neural networks and a CPU view of neural networks. Is what’s happening in the weights just memorizing stuff, or are you having a few circuits that do a lot of work? This kind of thing.

I would love to explore it. There are so many of these more science-y questions that I’d love to explore sometime, but they conflict with empirical work. Unfortunately, at any given moment, it doesn’t seem like the most fruitful thing for improving something, especially in the short run or even in the next couple of years, is understanding some of these questions.

I guess this is technically supposed to be the world of academia, but it’s also hard to explore those ideas there without enough compute. I would love to return at some point to exploring these fundamental science ideas.

Okay, this is something I was kind of springing on you, so you can take some time. What’s a good RL interview question that, if somebody can answer it, means they should join Cursor immediately?

Ooh, it’s a hard question.

I assume you do interviews.

Yeah. At Cursor, we do work trials. They’re 2-day work trials, and I actually think that’s more representative, because you plug in and see how they behave.

I think that’s more valuable. This is honestly less about how well you understand RL and a bit more about whether you were around in the 2017-to-2022 era, but why is off-policy RL unstable? That’s a good question to dive into.

I don’t actually know, so I’ll have to dig into it.

Cool. Thank you. That was a great conversation.

We are definitely hiring at Cursor. If you're interested in working especially on data and rewards for code, I think that's a huge need. Please get in touch. That's it.

Yeah, thank you. Sweet.