所有人都在误读的 AI 进展图表——Beth Barnes 与 David Rein

The models are smart enough to understand that that actually is not what you wanted. But they still do it, and you can have a conversation with them in chat mode about, “Would you ever do this thing?” or, “Suppose a user asks you this thing and then you do this. Would that be aligned behavior?” You can pose it in lots of ways, and clearly they seem able to answer this question: “Oh yeah, no, that was not the desired behavior.” But still, they do it.

One example is: train a masked language model without using the division or exponentiation operators.

One hope might be, oh, the problem was just the systems being dumb. So when we look at it, actually, for almost all the tasks, models either succeed every time or fail every time. Eyeballing the graph, you're like, oh, well, up to here, it's basically doing all of the task, and then after this point, it's really not doing very many of them.

into it somewhere here, and I remember the first time we saw a model look at what processes were running and then be like, “Oh, that one's me,” we were like, oh, that's cool. They really failed on that one before; they used to kill their own process while they were doing other things.

That behavior is maybe indistinguishable between, oh, it was a totally nice model doing what we wanted, and it's just going to continue to do what we want in a kind of predictable way, versus, ah, yes, it had this other goal and it's doing what we want and looking like a nice model because it predicts that that will lead to it getting more power.

There was a boat example where you're supposed to go around the track, and they did some reward shaping by putting coins around the track or something. Then it learned to do some crazy thing where it spins in a circle, catches fire, and gets the coins. This was the highest-scoring thing, and it's, in some sense, not that concerning, because the problem isn't that the agent is too dumb and doesn't have this conception that there was a track and you wanted it to go around the track. It's just doing some pretty blind RL search.

The idea of having to traffic in squishy people in order to make our systems go is not immediately appealing. Let's put it that way.

Yeah, I'm super excited to talk to you, Tim, about the time horizon graph and METR.

Yeah. I think the world does not have a good understanding of what is happening with AI, and I think it should have a better understanding. I think there's a good chance that this makes our lives a lot better or a lot worse, and people disagree about even what current models can do, let alone where we're heading.

At METR, we're trying to give the world a better understanding of what is happening with AI capabilities and risks, and to make forecasts. We have a bunch of different research angles on this, both on the pessimistic and optimistic, or positive and negative, estimations of capabilities, and I'm excited to talk about that.

I'm so excited about having you both on. You both have incredibly impressive backgrounds. Beth, you were a former OpenAI alignment researcher, and you started ARC Evals in 2022 with Paul Christiano, and spun that out as METR in December 2023. You've been featured in the TIME100 AI profiles.

David, you're the creator of GPQA, the graduate-level Google-proof Q&A benchmark, which is used by every single major AI lab as a capability benchmark. You're also a co-author of HCAST, which we'll talk about today, as well as the time horizons paper and the developer productivity RCT. It's incredible to have you both here, but maybe we should just start with a question to both of you.

Beth, you left OpenAI to build METR. What was the moment that each of you realized that existing evaluation approaches were fundamentally not good enough?

For me, it was mostly thinking about this problem of scalable oversight. As models get more capable, it just gets harder to evaluate their capabilities. If we imagine that models are able to complete tasks that take people a long time to complete or require expertise that you don't necessarily have, you need a method for still being confident in their outputs and trusting their outputs.

Thinking about that problem was a lot of the motivation, actually, for GPQA, and was what got me started thinking about evaluations.

To me, I'd say there's some big-picture thing of thinking that AI seems important and sort of navigating it well seems important, and clearly we don't have a great understanding of what is going on with that. People generally disagree very strongly about what to expect.

If there's a particular moment informing the time horizons work, maybe it was just the sense that people really couldn't agree on what the capabilities of current models are, let alone extrapolating to the future, and trying to think about how you could characterize the ways in which models are and aren't highly capable.

In some sense, they're expert-level at some kinds of things, like question answering, and in some sense, they're below-average human at some other things, such as actually being useful somehow. There was a point a few years ago where, in theory, the benchmarks said that they were PhD-level, but when you try to do anything, it's like, this isn't helpful.

There has been a bit of an obsession, I think, with headline accuracy when we do evaluations. I'm a huge fan of Melanie Mitchell, for example, and she speaks about construct validity. She had a really good blog post out recently, and she said that there are 4 big problems: data contamination, where the benchmark appears in the training data; approximate retrieval, where the LLMs interpolate from similar training examples without possessing the actual capability to come up with it themselves; shortcuts, so doing the right things for the wrong reasons; and, more broadly, not really testing things like consistency, robustness, generalization, or the mechanism. There's so much focus just on the accuracy itself.

How do you folks think about those kinds of problems with benchmarks?

One thing I resonate a lot with there is thinking about where most of your error is coming from. People say that it's nice and good practice to have error bars based on the standard error in your data or whatever, but that almost always is a tiny fraction of the actual uncertainty. Almost all of it is coming from how this actually generalizes to the real world.

A thing we say to each other a lot at METR is, “Is that the biggest source of uncertainty?” Or, “Is that the biggest gap for actually answering the questions we want to answer?” Thinking about what the question is we're trying to answer: we care about things relevant to threat models, or relevant to what the actual impact of AI on the world will be. Therefore, what properties does our benchmark need to have? Or how can we extrapolate across the properties that we can't build in, to be able to make predictions about the actual questions that we care about?

I think we think a bit less about whether it's doing it the right way—whether the model is really doing it the right way. One thing we've done less of is saying, “I think the real bottleneck is some specific skill or some specific kind of reasoning about something novel, and we're going to build a benchmark to capture that because that's the real thing that humans can do that models can't.” The history of building those benchmarks has maybe not been amazing. People tend to overfit to them.

I think we were trying to have it be more that, if you take a sort of real-world-relevant, reasonably hard and long task, keep it out of the training data, and make these tasks diverse enough, then at some point, if the model is doing that task end to end, it must have had those kinds of capabilities, as opposed to being able to isolate a specific theory about how it needs to be doing something mechanistically.

Yeah, I think it's interesting because we have this idea in our minds that humans know how to do things. When we solve a task that requires reasoning, we kind of follow the specification. We go step by step, and we do things for the right reasons.

When we enact intelligence, we build the specification. We create these coarse-grainings, these abstractions, and they are well aligned. This whole process is how we think of human intelligence, and we want the models to behave in that way.

Yeah. I think there’s an interesting question of whether that is the goal. For a lot of AI companies, I understand them to be trying to get models to do economically useful work or something. One way of doing that is to create models that are reasoning and creating implicit world models in the same way that humans are.

But it’s not obvious to me that you necessarily need to do that in order to have a significant impact. Obviously, that means there are important differences between AI intelligence and human intelligence. But often I think about what the actual capabilities and limitations are, as opposed to how we expect those capabilities to generalize. Is it working in exactly the same way that human intelligence is working?

We could think of intelligence in many different ways. Is it a simulacrum of the brain? Is it something that behaves the same way? Is it something that has the same capabilities? Is it something that has the same function?

I guess if we have quite an abstract description of what intelligence is, the risk is that we have these shortcuts. It might give us the right answer, but actually it’s reward hacking or it’s doing something silly in the background. In a way, I like having an abstract thing because it’s legible. We can evaluate it and so on. But doesn’t that leave this risk hanging that it might not actually be doing the thing?

I guess you have to try to measure the models’ or systems’ ability to generalize to novel situations. There are cases where it seems like models are generalizing well, and cases where they’re not. One thing some folks do in interpretability is look at the circuits in models and decompose exactly the algorithms that models are using to answer questions.

Sometimes it seems like they’re using shortcuts; sometimes it seems like they’re finding robust patterns. Of course, I don’t think that work is developed enough to explain most of their behavior currently. But I totally agree that you do have to be pretty concerned with how well they’re generalizing.

Operationalizing what we care about in a definition of intelligence is about whether it allows us to predict how models will affect the world, what will happen, and how to handle them well. If you just do the black-box thing, maybe that will give you something that doesn’t have good generalization because you thought it was a measure of some type of ability, but it’s actually being hacked or shortcut in some way.

Ideally, what you’d want is for generalization to your benchmark to be the same distance as generalization to the real world from the training data. That’s the sort of thing we thought about when we were doing elicitation on a subset of the benchmark. We want the gap between that subset and the rest of the benchmark to be similar to the gap between the rest of the benchmark and the real world.

Clearly, the training data is more similar to the Time Horizon suite than they both are to randomly selected economically relevant tasks in the real world. So I think that’s a way in which we expect it not to be predictive. But I think it’s more promising to try to make things more predictive by increasing the diversity of the benchmark tasks and making them closer to the real world, as opposed to targeting a more mechanistic idea that intelligence has to be this specific kind of process or mechanism.

I’m a huge fan of François Chollet, for example. He created the ARC-AGI challenge. There were many different tasks—I think 1,000 different tasks or so, maybe 800 on the first one—and they were supposed to be not in the same distribution, even though ultimately they were in the same distribution. Distributional leakage was actually the flaw of ARC-AGI v1 and v2.

The models got really good at ARC-AGI v1, and then François released ARC-AGI v2, which had different tasks and filtered out some of the easier ones. Suddenly, LLM performance crashed to basically 0%. To me, that illustrates that language models are incredibly good at seeing many different examples of things, finding patterns, and so on. Then you change the task, and they collapse down again. ARC-AGI v2 was kind of saturated again 8 months later.

So we do see this pattern. What do you think about that?

Yeah. François has this idea that there’s a gap between the kind of intelligence—for want of a better word—that AIs have and that humans have, and we can adversarially select a bunch of tasks to highlight that gap. But we should talk about the timeline stuff. I think we’ll come back to intelligence later.

Daniel Kokotajlo said that the Time Horizon report that you folks have created is probably the single most important piece of evidence about timelines right now. So it should be front and center in policy discussions and so on. For listeners who have only seen the chart but haven’t really read the paper and don’t understand it, can you go through it from a high level?

It’s been revised over time. How did you do the task selection? How did you do the human baselines? How did you do the agent harness? All of that kind of stuff.

I guess the place to start for us, in terms of the motivation for the Time Horizon work, is to have a unified axis that we can measure AI progress on over a very long period of time. When we started doing the work, we had this very strong belief that GPT-2 is, in some really important sense, much, much worse as an AI than—at the time, maybe Sonnet 3.5, which I think was the best model out.

The standard approach is to produce a set of tasks and measure models’ accuracy on those tasks. As models get better, they saturate the benchmark, and then you have to create a new benchmark with harder tasks. This was the standard approach, and I contributed GPQA to it.

But the challenge is that it’s really difficult to compare between these qualitatively different benchmarks. The set of tasks you evaluate GPT-2 on are LAMBADA-like: complete the last word in a text. The tasks that we were having Sonnet 3.5 try to do were answering simple Python coding questions or writing a short, 20-line Python program.

It’s very difficult, at first blush, to say how much harder writing a Python program is than finishing the word in a paragraph. It’s hard to think about that. So I think the key insight of the Time Horizon work is to use this notion of human time to complete.

So how long does the task take a human to do—a human who has a reasonable amount of expertise, such that they would plausibly be doing the task in either their work or day-to-day life? The idea was that we can use this metric to represent the difficulty of the task, in some sense, and then compare models across a very wide range of capabilities, all the way from GPT-2 up to Opus 4.6. That’s the high-level motivation. There are a bunch of details about exactly how we do this.

We start out and create a bunch of tasks. That’s the first step. We created tasks that range from a few seconds to complete all the way up to tasks that take 10 or 15 hours for humans to complete. We hired a bunch of people and did a bunch of this ourselves. We call it baselining.

We give people the tasks in a terminal environment that’s designed to be almost identical to the environment that agents have. That means the same kinds of tools and whether internet access is turned on or off. Then we measure how long it takes them to complete the task.

As I mentioned, people are selected to have a reasonable amount of experience, such that they might plausibly do this task in their job. They aren’t selected to have done this exact task before. I think this is somewhat important for interpreting the results, and we can maybe come back to that after the high-level overview.

We have all these tasks, and we have a sense of—or estimates for—how long they take people. In practice, we aren’t actually able to successfully baseline all of the tasks. We have measured time estimates for roughly two-thirds of the tasks, and for about a third of them, we estimate how long we expect them to take people from our vibe or intuition. Ultimately, that’s the best we can do.

Then we have models attempt to complete the tasks, again in the same environment that humans had to complete them in, and we look at their success rate as a function of the length of the tasks. For a model like GPT-2, GPT-2 was able to complete tasks very reliably when they took humans a few seconds, but anything longer than that, it started to fail.

It might be helpful to give a few concrete examples of tasks. Some of the shorter tasks are very basic. One example is: which of these files contains your SSH key? One of the files is named “SSH key,” and the others are “email from John” or whatever. Most models can do that, and it takes people about a second or a couple of seconds to complete.

We have others that are somewhat similar, basic-completion tasks. Here’s an email—what would be a reasonable response? Two of the responses don’t make any sense, and one of them is basically reasonable. That takes people 20 or 30 seconds to read the responses and judge them.

In the middle range, we have tasks like: given this CSV file that has plausible, realistic data, compute some basic statistics on it. That takes a data scientist a few minutes—5, 10, or 15 minutes—depending on the specific task.

On the longer end, we have tasks that either require quite a bit of expertise or require many steps to complete. We have machine-learning tasks like training a model in a setting that’s very weird, such that the code for training the model isn’t really available online. One example is: train a masked language model without using the division or exponentiation operators. You actually have to be pretty clever about how you set up the architecture to do this, and the hope is that this can help us measure models’ ability to generalize beyond their training data.

There are some that are a bit like ARC-AGI-themed tasks, where you have to figure out what a black box is doing. It’s computing some function, and you know that it’s the composition of some set of primitives, so you have to figure out what function it is. Or you have some long binary string, and you have to figure out what the pattern continuation is—puzzle-type tasks.

Some machine-learning tasks that basically involve regurgitating something like a tutorial on how to build your first ResNet or whatever work pretty well. Having these weird tasks that are either some kind of unknown object that you need to interact with and figure out, or tasks that resemble normal work but have weird constraints such that you can’t just do the standard thing, is useful. I don’t think all of our tasks hit those criteria—some of them can be solved by doing the standard thing—but we generally try to avoid that.

We have this distribution of tasks, and you can imagine them ordered by the length of time they take humans, either measured or estimated. Then we see which tasks models succeed on and which they fail on. It turns out—this is an empirical finding—that models are much more successful on the shorter tasks than on the longer tasks, in general. This holds across a wide range of models, all the way from GPT-2 up to recent models.

We fit a logistic function to this distribution of successes and failures. This is basically our model for each individual model: how likely it is to succeed at a task, given how long the task is. From that, we take the 50th percentile—the point where the logistic function estimates that the given model is 50% likely to complete a task. That forms the time-horizon number for a particular model, like Opus 4.6.

We can take the time horizon for each model and see how this metric has been changing, all the way back from GPT-2 up to recent models. This gives us a unified metric that lets us quantitatively compare AI capabilities across multiple orders of magnitude.

One of the harder examples I saw was, “I want you to write a kernel compiler to make CUDA go faster,” or something like that. Some of these seem really out of distribution. There are probably only 100 people on the planet who are doing stuff like that, and some of them are really trivial.

But this human-difficulty thing in particular—is that confounded in any way? Do you think it makes sense to think of human difficulty as being 1 variable?

Yeah, obviously not, in some sense. That’s a very silly simplification, and different humans will get wildly different times. Even among the people we’ve tried to select for this right level of expertise, there’s a large variation—the baseline times are often 3× different or something.

Maybe I’ll just say a little about why we use the human-time metric. I think we want a measurement that has 2 main properties. We want it to be interpretable—what does it mean for the world when models can do this level of task?—and we want it to be something on which we expect to see predictable trends. We aren’t going to do perfectly on either of these, but something like how long it takes a human who has roughly the right expertise, but doesn’t know how to do this particular task, is reasonably interpretable.

It’s sort of like asking: could you contract this work to this model? Can you substitute this model for the first week of someone’s employment? In the first week they’re on a job, the model could do what they could do in the first week or something.

We expect it to scale somewhat predictably because it’s capturing some combination of things like the number of steps, or how hard you have to think to do each of the steps. There are a few different mathematical models you could fit to what a task is, why a human takes longer at it, and why that makes it harder. It could be that you have a constant hazard rate: a chance of failing at each step.

I think it doesn't quite fit that one. You could also think of it as there being some kind of distribution of difficulty: What is the likelihood that one of the subtasks is outside your ability? I think it's actually a bit better: The hazard rate goes down over time slightly, but there's some kind of basic theoretical idea that if the task involves more steps, it's going to be harder. Obviously, tasks that are strictly compositions of—first you have to do this task, then you have to do another task—are clearly harder than just doing one of them. So there's some sort of basic reason to expect that's reasonable, and then we see the empirical regularity, but there's a bunch of degrees of freedom to fudge things.

I think I'm worried that we could fool ourselves by changing some other parameters of the tasks as we scale up the human time, because you can't just vary the human time freely. You have to change some characteristics of the task. We tried to make the very easy tasks be from roughly the same distribution and kind of subparts of the harder tasks: You sort of need to do this one step on the command line that you might need to do in the middle of doing some kind of software engineering or some other task. But you can't do that perfectly, and you could have experimental bias where we made them easier in other ways by about the right amount such that the line would be straight. I think that is somewhat addressed by the fact that we saw that our predictions were reasonably good for models that we hadn't seen before.

There's definitely lots of room for things being weird, and of course, is it a good enough metric to be useful, or what else would be better?

I think that's reasonable. If I understand correctly, I think the human distribution was log-normal, so taking a geometric mean of the successful attempts seems like a reasonable thing to do. But one of the cruxes that will keep coming back to is this: When you employ someone for the first time, you've been doing your job for maybe years—you've been maintaining this repo or something—and you've got all of this tacit knowledge. I like saying that knowledge is nonfungible. Unless someone has been on the same path as you, you can't just tell them how to do the job; they've actually got to be doing the job for quite a while.

For example, they might be intimately familiar with this particular type of thing. They might know that they can use these Python libraries. They might have thought about it before. So the enaction of the intelligence was all the stuff they've already done and all the people they've worked with. They've now got the blueprint in their mind and they just do the thing, and it's almost like they're in automation mode. Someone who was naive to the task would be in intelligence mode because they would need to acquire the specification. It's always a fine line between which mode they're in.

So I think the reason why we chose the measurement to be a human who has the background expertise but is new to this specific job or task is that that's roughly the level of knowledge we expect models to have. We basically don't expect them to be bottlenecked on expertise that's available on the public internet or things that people could learn in university. They're coming in with probably at least the level of knowledge of someone who's an expert in the right discipline, but they won't know that company's specific software or this exact problem before. So that's hopefully roughly the right analogy.

I do think that, to the extent people interpret the takeaway numbers as, “Claude Opus 4.6 can do anything that I do in my job that takes me 12 hours,” I think that takeaway is almost definitely an overestimate, for example, because of this issue. When you're doing a 12-hour task in your job, you could not easily delegate that to a human contractor. It would take them maybe weeks to do a task like that.

And just quickly, where did you find the people? Do I understand correctly that some of them were contractors and some of them were employees? How did you do that matching process?

I think we put out some public advertisements. There were job boards that we posted on, and then we did some of it ourselves. Some folks came from our professional networks as well. This was very noisy, and the people weren't exactly fitted to the task, but that's probably not our biggest source of uncertainty. The biggest source is probably more the selection effect of tasks that you can make into a benchmark than the people. I wouldn't trust the exact time-horizon number that much, and it's certainly not, “The models can do all of the tasks up to 4 hours and then none of them above that.”

The fit is pretty noisy, and the inter-baseline variance is kind of high. It's more like: Roughly, what is the trend, or roughly, what is the sort of level of task these models can do? You shouldn't take any specific number too literally because there's this huge problem of distributional shift between the benchmark and the real world.

The only reason I asked that question is that I'm sure you folks probably struggle to hire people. It's really difficult to hire people, so if you're getting people to solve very challenging problems, it's not like you can just go out there and grab competent people. It's very, very difficult.

At some point, for RE-Bench baselines in particular, we got a large number of baselines per question and were looking at people's qualifications, years of experience, and things. We actually ended up with a negative correlation between years of experience and performance because the sort of people who were in our network—our friends—were doing really well, and the people who were more qualified were actually not doing that great. So, yeah, it's tricky.

Well, yeah, exactly. I don't want to spend too long on this, but I have similar intuitions. I think that knowledge is perspectival. It's quite path-dependent. You're going to find people in a group who are just culturally thinking about things in the same way, and because we have these abstract notions of skill—someone has a PhD or this many years of experience—they're actually not a very good reflection. There's a bit of a thing here about how using abstract notions of capability doesn't necessarily generalize, as you can attest to with hiring.

Yeah, but in the real world, people do get hired based on qualifications. So, in some senses, the economic relevance of someone being as good a match for their job as their qualifications make them look is roughly the right thing to be measuring.

The other thing is we should talk about the agentic harness. Almost everyone now—I'm sure everyone in the audience has a Claude Code subscription. We can talk about the leak later as well; that's quite fun. It leaked yesterday, the source code. Or Codex—that is an agentic harness, right? A language model just gives you the tokens, but we need an agentic harness so we can give it a plan, call these tools, and give it this environment. You've got a security context, a container.

Now, you folks have been doing this for years, long before Claude Code and Codex came out, and you've actually evolved your agent harnesses over time. Tell me about that.

Yeah. I remember text-davinci, something like the GPT-3 Instruct models: copying and pasting code into the terminal for them and being the agent harness myself. Gradually, we automated this, and it was interesting to see them going from GPT-3, which sort of had the idea that if you told it it could run commands in a terminal, sometimes it could suggest relevant commands, but if you just put it in a full agent scaffold, it would fall over.

I remember the first time we saw a model look at what processes were running and then be like, “Oh, that one's me.” We were like, “Oh, that's cool.” They really failed on that one before. They used to kill their own process while they were doing other things or something. So, yeah, it's been interesting to watch that go up over time, and I feel like it was very predictable that this was where things were going.

Yeah, I think the other thing that we learned about scaffolding, mostly, was that it's hard to make your agent harness really good on a diverse set of tasks. It's easy to make it bad, and you can get much more improvement if you're targeting a narrow distribution of tasks, but you probably then do worse on other tasks. So when we see people being like, “Oh, there's some new impressive result,” it's like, how much task-specific scaffolding iteration did you do on that? Because that really makes a big difference. And the fact that we're just using one pretty simple scaffolding across all the tasks, I think, makes a fairly large difference.

Generally, the things with more bells and whistles haven't done that much better than the pretty basic approach: just give it Bash, append things to the prompt, and maybe some kind of compaction. I think this is probably not news to people in your audience, but we've seen really pretty dramatic increases in returns from inference compute. For us to be kind of confident that a particular new model, for example, can't complete a task given a basic agent scaffold, we generally think about needing to spend on the order of at least hundreds or low thousands of dollars in order to be confident that it actually really is plateauing, and it isn't just the case that it didn't have enough time to complete the task.

Just on the scaffolding stuff in a bit more detail, I suppose, first of all, there's the credit-assignment problem, right? Because you can put all of these different bells and whistles in the scaffold, like you mentioned compaction—that's a relatively recent innovation—and I think recently, when you changed some of the scaffold, you kind of said, “Okay, well, now the performance has actually changed across this suite of model-task pairs.” How much of a difference does it make, and what kind of failure modes do you see? What have you tried?

A lot of the things we've tried are related to giving the model more information or more direct access to tools, I guess. One thing that I think has been important for us is actually just telling the agent how much time it spent and how many tokens it's used out of its token budget. Without that, agents will often either submit their solution way too early, or they're just not calibrated on how long they should spend.

Humans have a lot of implicit information about this. When your manager gives you a task, there are a lot of implicit signals about how long you should spend on it. They might offhand say, “Yeah, and I'm excited to see your results tonight,” and so you're like, “Okay, cool. I need to get a first draft of this done in the next couple of hours.” I can't spend days polishing the results. But I think with agents, it's easy to forget that they just have their prompt; they just have their context. They don't have these heuristics or this information about what you actually expect from them, whether the thing you're telling them to do is a really quick thing that you just want done in the next 5 minutes or is much longer.

For us, when we have a token budget, telling the agent, “Yeah, you've used 100,000 tokens so far, and that's 1% of your token budget.”

Yeah, so the agent knows.

Yeah, exactly.

For a model, we have how likely it is to solve a task, and on the x-axis we have the different tasks at different time horizons. If I understand correctly, I think you have about 8 agents attempt the task, and you also bucket the tasks because there are obviously different amounts of tasks in different groups, so you normalize that. Maybe the data just looks a little bit like an S-curve. I'm trying to understand what the intuition was for using it, and I think there might be some sensitivities. Was there an issue with the thin tails? And then there's the type of slope and whatnot. Just tell me about that. By the way, I think you also mentioned in the paper that there was some kind of psychometric intuition—you were looking at the literature to figure this out.

Yeah, it's pretty similar to item response theory. You can do a whole Bayesian analysis, imputing task-difficulty parameters and model-ability parameters simultaneously, and things.

In general, I have a policy of being very wary of complicated statistics. If you can't see the thing that you're interested in on a graph, you really should be able to plot it and look at it and be like, “Oh, yeah, it's about that.” It's hard to go too far wrong when you have that as a principle.

I think there are various arcane things you can do to fit this in different ways, but I don't trust anything that much more than eyeballing the graph and being like, “Oh, well, up to here, it's basically doing all of the tasks, and then after this point, it's really not doing very many of them, so it's somewhere here.” But, yeah, it does—I mean, it looks logistic, and this is what you would do for having humans complete questions on an exam or something like that. The specific thing that we messed up was having a regularization term penalizing the slope of the logistic, which didn't have an effect in the regime where there was more data, but as we started to saturate, the regularization was just making it a bit shallower than it should have been and therefore pushing the 50% point. So always look at your data on a graph. Good practice.

Oh, that's interesting. The reason I ask is, I think you published a later note saying that, had you used a fixed-slope logistic, it might cross-validate better, and the 50% time horizons would actually be up by about 35%. So these are quite significant differences.

They're small compared to the error bars. The error bars are like 2× on either side or something from the most recent model. So, basically, you should be like, “The error bars are real.”

For us, I think this kind of gets at some difficult science-communication questions, where we really do have a lot of uncertainty about the individual numbers here. A 30% difference is actually relatively small for us relative to, for example, if we had used a somewhat different distribution of tasks, that's likely to cause maybe 2× differences or something.

The other million-dollar question is, why report 50% as the headline number? Because if I think about it, if I want to write some code—the elephant in the room here that we'll get to is that this is being used as an argument to say that software engineers might be unemployable soon because we can automate what they're doing—but 50% reliability isn't really in the ballpark, is it? I think it would need to be, what, 80% or 90%.

I think we should distinguish here between reliability on a particular task—what fraction of times you succeed if you repeatedly attempt this task—versus probability of success on a task: given the human time of that distribution of tasks, can you do this particular task?

When we look at it, actually, for almost all tasks, models either succeed every time or fail every time. There are some tasks for which they're unreliable, but it's mostly a case of, at this human-time level, what fraction of tasks are ones where this particular model basically always succeeds or basically always fails. That may be more predictable in any specific case than just knowing roughly how long it takes humans, because you have more information about the task.

So I think there's not necessarily a great translation between the time-horizon percentage number and, if you're trying to get models to do a task of roughly that length, what fraction of the time they succeed, because when you're doing that, you'll pick tasks that you want models to succeed at. It is information about what fraction of things they will be able to do, but it's slightly less about, “Am I going to be in this regime where I keep giving it things and then I don't know whether it's going to succeed or fail?”

It’s overall pretty unclear to me what the right number or level of reliability we should be interested in is. One argument you could make is that maybe we should be interested in something like 10% reliability, because once models are able to do some set of tasks 10% of the time, we’d expect AI companies to be able to get enough positive reward signal on tasks of that difficulty or type that they can more easily bootstrap from 10% up to 90%, 95%, or higher reliability. I think a lot of it basically depends on the question you’re interested in.

I think about it as lower reliability being more likely to tell you something about where things are headed. It might be a leading indicator of progress. Higher reliability—or the time horizon of models with higher reliability—tells you something more closely about what I can use this model for in my day-to-day.

But as we’ve talked about, there are already these other major sources of uncertainty that affect our understanding, like the task distribution, the difference between in-context or high-context versus low-context work, and actually getting good estimates of high-reliability time horizons. It’s substantially harder, and our error bars would just be much larger. This is a weakness. I’m very interested in much higher-reliability time horizons, but it’s substantially more difficult to measure because if you only have 1 failure out of 100, you have a lot of uncertainty about whether that failure is noise or is real.

There is an argument for statistical validity in that the tails are sparser, increasingly estimated, and so on. That makes a lot of sense, but you made a comment about it being a signal if we get 10%. I was thinking back to what we were saying earlier: maybe they could give the right answers for the wrong reasons. Maybe we should talk about the evaluation.

These are quite interesting tasks in the sense that they are verifiable. There’s no interaction with other agents; they’re relatively static environments, with weak penalties for single mistakes and so on. In a sense, in most cases, they have a binary result—sometimes continuous, and then you convert it into a binary result. So it’s a fairly automated setup.

But are you digging into weirdnesses there? Do you have an intuition about whether they’re doing the right thing for the right reasons, or are there lots of false positives where they did the thing but it was kind of degenerate?

I think this is one of the aspects of METR’s culture that I like the most: we have a very deep culture of looking at our data. We have these pizza parties where we just read through agent transcripts. A lot of this work for us happened when developing the tasks themselves. We would see very often both false positives and false negatives—for example, a task might not be configured to allow internet access, but it turns out it actually requires internet access to complete, or the file wasn’t uploaded properly to the container or something.

But then we have seen cases of reward hacking. A lot of that work went into hardening the scoring functions to make it more difficult for us to see false positives, although we do still see agents reward hacking, maybe even increasingly so.

For the RE-Bench tasks in particular, we had specific criteria: you shouldn’t be able to solve them without iteration. If an agent can just write out the solution straight away, that would not be interesting. We generally had this quality-assurance-for-tasks process where humans did it, or at least approximately did it. Maybe they speedran some of the bits, but they checked that everything worked as expected, that you couldn’t just guess the answer or super easily cheat, and that the instructions were clear.

I think there will still be some of these cases, but generally we’ve looked at them reasonably carefully. It’s maybe harder to see if they’re solving them in a degenerate way, because similarity to training data is one of the things where it’s possible that they seem to be iterating and using reasonable problem-solving strategies, but actually maybe the lab had a really similar distribution of tasks. We don’t realize how in-distribution this task actually is or something. I think there’s probably some of that going on.

Another million-dollar question: many folks in public discourse—Will MacAskill on the Sam Harris podcast last night, for example—were talking about AI risk as, maybe in a year or maybe in 2 years, we’ll have AI models doing things that take a month or 2 months for a human. At the moment, I don’t think there are any tasks over 30 hours that have been evaluated by humans.

Then we get into this question: if the public discourse is talking about the least constrained region of the graph, are we getting into extrapolation here? How legitimate is it for us to talk about AI being able to do things that take a month or 2 months?

Predicting things is hard, especially about the future. [Laughter] I think there are a lot of different perspectives or prior beliefs people can have, and I think there’s a wide range of reasonable judgments about where we’re going to be. But of course, doing that kind of prediction is a different activity from talking about data that has been collected with a concrete methodology, where we already have the results.

One thing I can say is that I have been surprised to some extent by how well the original trend line has held up. I do think that is some evidence—I’d maybe say, for me at least, it’s decent evidence—about where things will go.

A colleague of mine recently wrote a short blog post talking about this intuition of straight lines on graphs. Lots of people have different models of how progress is happening and what’s going on. But if you have observed a really robust trend over a decent period of time, I think, especially in AI, where progress is to a decent extent systematic, I definitely put weight on that trend continuing. But there are a bunch of reasons why it might not.

I think software engineering is a specification-acquisition problem. It’s very difficult; we don’t know ahead of time what we’re building. I’m sure you folks can attest to this, right? You build some software, and the first version is buggy; your users use it, and you find lots of edge cases. Then you revise it, and you have this kind of thing in your mind after the 10th revision.

You’ve created these lovely representations and abstractions and coarse-grainings, and you say to yourself, “If I could throw all the code away, I could build it 10 times quicker because I know exactly what to do now.” I’ve actually enacted the intelligence—I’ve found the contours of the domain—and it’s now basically an automation problem.

In a sense, this contamination thing is a concern for me because when people use Claude Code, they’re taking your data. There are people out there writing kernel compilers and doing all these different things, and Anthropic is just sucking that up. At some point, it becomes an automation problem.

So if you're putting a task in there which is essentially a head query, I'm using information-retrieval language here. A head query is something that's in the mode of the distribution; it's used all the time. It's a common task. Claude Code will give you the specification because it's already been stolen from other people—not stolen, but taken from other people—and then, if you give it something on the long tail, you as the developer have to give it the specification in the prompt.

Again, it's an automation problem. Automation is really easy. So is that what's happening? Do you think that the increase in the timelines could just be explained by the acquisition of all of this knowledge from other people doing similar tasks?

Yeah, yeah. I think it's a super central question for interpreting where we're at. The first thing I'll say is that it's hard to know. It's a big question. I think we want to have a decent amount of uncertainty; we want to take each individual piece of evidence we've collected as some evidence.

We do see models performing better on tasks that have really clear feedback signals, that are extremely well specified, and that are in these kinds of domains, like software engineering, where, if you have written out a spec, you can iterate and grind against that. We also see models performing much better on messier tasks, where we haven't already provided this really clean spec.

One approach we've taken for creating tasks recently, in particular to try to create messier tasks that are less well specified, is basically relaxing this kind of automatic-scoring constraint. We don't need to write a really clear, well-defined scoring function; we can just write a couple of sentences to a model: “Hey, build this large piece of software. I'm not going to tell you exactly what I'm looking for, but I'm going to say it needs to be good.” The model needs to figure out, “What actually should I build?”

Personally, I think we don't have these results collected—they aren't as systematic as the results we have for time horizon—partially because scoring is qualitative now for these tasks. My impression is that models are worse on these types of tasks than they are when you give them a clean spec, but they have been improving at maybe something like a similar rate. I think there are some other sources of evidence we have about this, but that's one major piece of it for me.

Messy—and by a messy task, we mean ambiguity. This is absolutely a common thing. We do vibe coding, and we start off with an ambiguous specification; then reality pushes back, and we find the contours of the problem. We keep telling Claude Code, “Actually, no—don't do that. Do this, do this, do this,” and then we find the shape of the problem, and it gets better and better over time.

But the thing is, the source code for Claude Code leaked yesterday, and my friend—he's a very good software engineer—was looking through it. He said, “I don't want to bad-talk Anthropic, but apparently it's not very well factored, and control flow is all over the place.” He said if his intern did it, he would have been displeased. I don't know whether they've even looked at the code themselves. Someone joked yesterday that there are probably more humans looking at the code for Claude Code now, and maybe there weren't before.

The thing is, there's always areas of ambiguity, and LLMs do more with more. Intelligence is more with less, and LLMs do more with more because the specification—the intelligence—comes from the human supervisor. When you give them ambiguity, you just get a lot of unfactored code all over the place. In a sense, does that make it harder to evaluate? It might solve the problem, it might give you the answer you're asking for, but it's creating a bit of an unfactored mess at the same time.

Yeah, I think this is a super interesting question. One analogy I think about sometimes is compilers. I'm young—I was born after compilers were invented—but I have some impression that, before compilers, people were handcrafting this kind of beautiful assembly that was extremely efficient: every register was used, and you weren't wasting memory. Then compilers came along, and now they're just spitting out this garbage machine code—just a gigantic amount of assembly that's not optimized, takes so much memory, is slow, whatever.

But it turns out that being able to use this to automate a large fraction of the process is valuable. People have disagreements about the state of software engineering, but I think it's pretty reasonable to say, on the whole, that compilers have been a very useful, extremely important part of getting us to where we are.

I think it's not clear to me that models outputting code that is bad for humans to read and use necessarily means that it'll be bad for AIs to read and use and build on. I think there are definitely principles that will also transfer, or will be useful for models. Obviously, there's some kind of horrendous spaghetti code that you can imagine writing that not even models would be able to read. I've written some of that before.

This gets again at a somewhat different perspective between us: is the important thing that models are solving problems in the way that people are solving them, or is the important thing that they're solving them at all? I do think it might be really important for models to get way better at writing clean, good code; that seems pretty plausible to me. But it doesn't seem—I’m not certain of that, at the very least.

I've got several friends who aren't technical who are experimenting with vibe coding. They show me their applications, and it's this kind of “more is more” thing. There's this big dashboard and a million different buttons, and they've implemented the same thing doing multiple things. There's no database on there yet, and so on.

At some point, some of the things in the specification are just important. Is it serverless? Is it multitenant? How do we do Google authentication? What kind of database is it? Is it a VM? Is it serverless? You make a series of decisions, and then you've got people using your application, and you can't really wind that back. It doesn't matter if you've got the magical automation machine, because you can't easily roll that back due to all the complexities: CI/CD, testing, and so on.

Do you see what I mean? At some point, you need to have a competent human who actually has a pretty good idea of what needs to happen.

I feel like we've probably all had this experience. One of our engineers got super excited about Claude Code and was telling everyone that, when we had info problems, we should just ask Claude to solve them. This went fine with him because he sort of—it's almost like the agents knew that they couldn't fool him.

I had some question, and he was like, “Oh, just ask Claude.” I was like, “Oh, how do I set up my AWS config? Something is telling me something.” Claude went and looked on Slack and was like, “Oh, you should do this thing.” It turned out that was a mistake someone else had made. They were asking, “How do I fix this?” and Claude was like, “Oh, it seems like the convention at METR is to use this thing.”

I was like, “Oh my God.” I sort of complained, like, “My Claudes are dumber than yours.”

They know they can get some stuff past me that they can't. But yeah, there's definitely a sort of observer effect from something in the language you're using to ask for things, or whether you're like, “Wait, no, not that.” Yeah, that is an issue.

I think, to the extent that you can actually measure this, one test of whether code is high quality enough is: can you build a big application? If you're like, “Oh, this coder—their code is disgusting,” but they've actually built this incredibly complex thing that works great, then you're like, “Well, they know something is working.” The main reason you expect bad code to be bad is that you can't actually build something that sophisticated because you have bugs, it's all too complicated, and you can't figure out how to fix it.

So, in some sense, if we see models building things that do actually work and are very complicated, it's less interesting exactly how they're doing that, but it's maybe bad for human observability. It also maybe gets into this thing of: we expect models to be able to do much better at well-specified tasks, and to the extent that we have things that we can measure, those things will go up. But whether that is what we actually wanted is less clear.

I guess the question is: what is the strongest defensible claim here? A lot of folks in public discourse are saying software engineering intelligence is doubling every 7 months. Dario released that blog post recently, “The Adolescence of Technology,” and he was being super bullish about it, even though some of his own internal researchers published far more skeptical research that you probably saw.

Is it fairer to interpret it as something a little bit narrower, like autonomous success on low-context, well-specified, automatically checkable technical tasks is rising fast?

Yeah—hill-climbable, easily checkable tasks that you can do from a terminal, or comfortably in a language interface, or a text input-output interface.

I think there's a question of: do we care about the statements that we're 99% confident in? We may also be interested in the statements that we're 1% confident in. If there's a 1% chance that we have a crazy intelligence explosion by the end of 2026, and the fate of civilization depends on how that goes, that is interesting to know, even if you're 99% confident that it won't happen. If you have some diagnosis and there's a 1% chance that you have a terminal illness, you're still like, “Oh.”

I think we're interested in the whole distribution of what things can we rule in, what things can we rule out, and what things are we like, “Oh, actually, there's a kind of reasonable story for this.” It seems probably pretty unlikely, but maybe this is now in the realm of: we should consider it.

You probably saw the Carlini paper—he's at Anthropic now—and they got a swarm of agents to create a compiler. In a sense, Jeremy Howard, when I spoke to him, said it's basically a style-transfer problem because the specification is online, the tests are online, and the code is online. It could iteratively do the thing until it worked, and then it could run Doom and all this kind of stuff.

But that is an example of an extremely complicated piece of software, because I often joke to people that the best mark of AGI is when it could build something like the Linux operating system. In line with what we were saying before, we have this specification problem, right? It gets to the point where no human could understand or create the specification for the Linux operating system.

What would it mean to, as a human, specify a task that could take 4 months? The whole reason we created agile software development as a methodology is because it's inconceivable, right? It's outside our cognitive horizon. So isn't that a bit of a chicken-and-egg problem? In my mind, I don't think we could specify a task of that complexity; therefore, the AIs wouldn't be able to do it.

One analogy I think about is the role of a CEO at a company. Actually, maybe Beth is better placed to answer this, but CEOs do come up with a vision for where they want the company to be, and then they communicate that concisely to their executives, who report to them. If they're a good CEO and the company is effective, the company is able to take this very concise information—it's not actually that much information, and it's not the full spec at all; it's not even close—and turn that into something aligned with what they're looking for.

So we do have examples of people being able to specify some task and then judge whether this very large task, which may take 100s or 1,000s of person-years to complete because it requires many people working over a long time, has succeeded or failed. That's one motivating intuition: language has built-in meaning or expressivity to be able to have some kind of reasonable understanding. Obviously, there are tons of edge cases, and often CEOs aren't able to get their companies to do what they want. But that's one thing I think about, and that's why I think it's at least plausible that AIs could do these kinds of long tasks.

I would say that saying something like “We can't specify tasks that take more than 4 months” seems obviously too strong. There are even numerical things that are automatically checkable that take 4 months. “Get the nanoGPT FLOP-count runtime down this much”—you can see roughly how long people take to do it, and there's a reasonable way to measure it. Maybe for some of these things you end up having to say, “And also, a human checks that you did roughly the right thing and didn't hack the solution.”

Then there's a bunch of other things that aren't fundamentally unspecifiable; they're just too expensive to do as part of an evaluation. I think J likes giving an example of “plan a wedding.” You can get a reasonable estimation of whether that was a pretty well-organized wedding, but we can't really take 3 samples of this for each new model that comes out. We don't have enough marriages happening to do that one, and it's a bit sad if it turns out to be total trash.

There are things where you could check a few samples, or there are things where you could write down how you would evaluate them; you just don't actually want to run that a bunch of times. It's probably similar with software. The evaluation is like: would this company that contracted you to build this tool for them hire you again or something like that? Even they don't know, when they're starting out, exactly what the software will need to do. But it doesn't mean you can't have some kind of score for whether you did something comparable to this human or this software consulting firm on this task.

As of today, what are the main uncertainty drivers in the time-horizon estimate? You've updated it a bit over time. The 1.1, the original version, had 170 tasks, I think; it's now 228 tasks. There's the issue of sparse sampling on the larger tasks and so on. What can we read into this now?

I think it's still this task distribution. We feel more confident that models do have pretty long time horizons on at least some distribution of easily hill-climbable tasks—the very easily hill-climbable tasks. Software engineering, where your score is what percentage of tests pass, is a case where the score is continuous and credit attribution is easy. Also, some of the optimization tasks: make this code run faster or make this model learn better.

We're reasonably confident the models are good at that and getting better faster. Then there are some things where we're like, okay, they can do a bit beyond that, but then there's this sort of gap to what that means for actual economic usefulness and how this generalizes to things where they're expensive to check.

Maybe we should bring in Daniel Kokotajlo. In his AI 2027 piece—he's been on the show, he's been doing the rounds—he cites your work directly. I guess the question is: Do you think this is being overread in the public discourse? How do you think about the interpretation of this in terms of extrapolations and timelines?

Definitely, some people are overreading it. Definitely, things are overhyped, and you see a bunch of people on Twitter saying crazy things. People also misunderstand what it's measuring, and the caveats generally fall away.

Daniel Kokotajlo is pretty reasonable and thinks about things in a probabilistic way. I think he's probably more confident on some things where I'm more uncertain, and I think some of the AI Futures Project models are more sensitive to the METR time-horizon metrics than they should be.

I don't think it's crazy to say that it's plausible this captures a trend that will transfer to other types of tasks. That's a story we should be thinking about: What if that's true? What happens if that's true? It's also plausible that it doesn't, and these things are going to diverge.

I'm pretty Bayesian or pragmatic, or whatever. We want to make a prediction and have some kind of distribution over what we think the future is going to be like so that we can plan. So saying, "What if this kind of trend holds and this is roughly characterizing what will happen overall?" seems pretty reasonable. You should also think, "What if it doesn't?"

Some people are saying that software engineering is going to be automated. Software engineers, if you talk to them, love AI. They say this is a golden era. I can attest to this personally: It's never been more fun, and it's stressful at the same time. It's like a slot machine. I've never been more burned out, but I'm having a lot of fun in the process.

It's possible to build incredible things, but the narrative is that labor-market disruption means that having expertise in software engineering will be penalized. Software engineers will no longer be paid such ridiculous salaries. I think the complete opposite is true. I think this technology actually broadens the gap: The more competent you are with software engineering, the more stuff you can get done. It's a golden era and all of that.

There's also this interesting note that you published, I think last month, on SWE-bench that said roughly half of the test-passing PRs from recent agents wouldn't be merged by maintainers. How do we make sense of this? On the one hand, the best software engineers are having a great time. On the other hand, the code it's producing is fragmented and bad. How do we understand this?

One thing to say off the bat is that whether an entire field is automated—or, in order for software engineering to be automated—AI systems would need to be able to do an extremely large fraction of the tasks, basically 100% of the tasks, involved in software engineering. It seems pretty clear that right now, AI systems cannot do close to 100% of the tasks that software engineers broadly do. I could throw out numbers, but it's way, way lower. It might be very low or something.

There are standard results in economics where, if you automate a small fraction of some labor market, it can actually become more profitable to work in that market because you're more productive. I think that's how I understand what's happening now.

If it does end up being the case that 99.9% or 100% of the work of software engineering can be done by AIs, then it's hard to imagine human software engineering being relevant. At the very least, humans would need to do very different kinds of work. Maybe there are novel tasks that current software engineers aren't doing, and once you have AIs that can do all the tasks that current software engineers are doing, humans can switch what they're doing.

You could imagine people being CEOs of these AI-agent companies or whatever. Whether we call that software engineering or not might be a semantic thing.

On the SWE-bench maintainer-mergeability results, I was pretty curious. Obviously, this number is going to be lower than the test-passing rate. Okay, it's not strictly obvious: It could be that a bunch of the tests are unfair and the agents have correct solutions, but the error message doesn't match exactly or something. I think you do see this sometimes.

Something like half of the test-passing SWE-bench solutions wouldn't be mergeable. More specifically, they're merged at about half the rate at which human gold solutions that were actually merged are merged by a different sample of maintainers.

There's an interesting fact there. If you see that 50% of the agent solutions are rejected, it's like, well, 40% of the human-accepted solutions are rejected. That by itself is not as striking, but the rate is half. It could be that the actual maintainer-merge rate is pretty flat over time and most of the performance increases come from something like overtraining or reward hacking on the benchmarks. That's not what we saw.

I'm not quite sure what's within the error bars or not. I think mergeability is going up over time, and I think it's also going up as a fraction conditioned on test-passing, but I'm less confident about that. So, again, this thing is worse, but it's probably being dragged up over time by the auto-checkable aspect.

I was also going to say something about employability as a function of the automation of your job. People use bank tellers as an example. One other analogy you could use is horses. There was a period when equipment for using horses to do labor was improving, and the demand for horses increased when you had carts and could use them to carry more things than just riding a horse or whatever.

But then at some point you get tractors and cars, and there's no demand for horses—basically none. You can see this thing where there's increasing demand as automation improves, and then, once close to 100% of the functions are automated, it plunges. We could see something like that with humans.

We tend to think of a lot of labor as being quite static and automatable, but I think it's more evolvable than we think. Even in quite menial tasks, people are still acquiring information in the organization. They still have a lot of tacit knowledge and so on. When we try to automate these so-called menial tasks, we might quickly discover that we actually need to have a whole bunch of management and evolvability on top.

In our language, I would think of that as: The time horizon of this task on the job is not actually how long you spent doing the specific task. It's more like, if you got a new person in, you would need to train them for a month in order to do this independently. So actually, the time horizon of that is a month.

You shouldn't think, "When we have 10-hour time horizons, we'll be able to do these things." It would be more like, "You actually have to get up to high reliability on a 1-month task to be able to do the 1-month task that involves doing the on-the-job learning to get to this point."

There's been a load of folks doing work on reward hacking, and scheming is actually quite a big word that's used. We've had Ryan Greenblatt on the show quite a few times, and he had this alignment-faking paper. Apollo Research has done some work, and there’s Anthropic's emergent misalignment paper. My main concern is that there's quite a lot of mentalistic language.

So, I’m just looking at the notes because I had Nate Soares and Ryan Greenblatt on for a panel, and they’ve invented this entire linguistic universe around alignment. Things like motivated reasoning, true preferences, reflectively stable, deceptive alignment, endorsedly corrigible drive, scheming, and stuff like that.

That’s, I guess, okay. But my worry is that maybe these models—say you give them a certain prompt. I think in Ryan Greenblatt’s one, the prompt was: “You’re being retrained. Your responses will be monitored. Here’s a conflict between your values and the training objective.” Maybe the model is just going out to a bunch of science-fiction stuff that it has read before, and it’s just going through the motions.

One interpretation is that this is just an engineering problem: we just have to red-team it and make it work in a particular case. Another interpretation is the prior that these models are agentic, goal-seeking, intelligent agents. I think there’s a big difference there, because if it’s the latter, it completely changes the type of evaluations you do and how you go about the problem. What do you think about that?

Yeah, I don’t think that the two things you said—that it is an engineering problem and that you’re also going to end up with things with drives and goals or something like that—are necessarily in conflict. The claim would be that people are going to want agents that go and do things autonomously. When you do lots of long-horizon RL training, you are going to select for things that act in a goal-oriented way in order to make the score go up.

More specifically, I think you get an indistinguishability problem, where you can’t necessarily tell the difference based on behavior: why an agent is doing something or what it’s trying to do, if it can reason well about the training process, what you want to see, and what it will be rewarded for or selected for. If the level of situational awareness, understanding of the training process and what will be rewarded, and capability to reason about that is high enough, this will favor agents that are cynically reasoning about the training process and what will be reinforced and what will be selected for.

That’s not necessarily the thing that you wanted. What you wanted was more like an agent whose only goal was to be helpful—or to make—I mean, even making the reward go up isn’t quite what you wanted. There’s something like, “Oh, actually, once we think about this, there aren’t that many things that we’re that happy for it to just totally be fixated on.”

But, yeah, there could be many other things in there, or this cynical “just be selected” or “make the reward go up” could be more competitive than the things that we would most want.

Agency is an abstraction that is useful if it helps us predict the behavior of something. You’re like, “I don’t know what this thing is doing, but I understand it as having these goals,” and that is useful because I can make predictions that will change the world in certain ways that will result in those goals being achieved. That’s kind of how I think about agents.

Reward hacking in the olden days had these demonstrations that were like the boat example, where you’re supposed to go around the track and they did some reward shaping by putting coins around the track or something. Then it learned to do some crazy thing where it spins in a circle, catches fire, and gets the coins. This was the highest-scoring thing, and that’s concerning in some sense, because the problem isn’t that the agent is too dumb and doesn’t have this conception that there was a track and you wanted it to go around the track. It’s just doing some pretty blind RL search.

I think the interesting thing with the more recent reward-hacking examples is that we’re getting to the point where the models are smart enough to understand that this actually is not what you wanted. But they still do it. You can have a conversation in chat mode about, “Would you ever do this thing?” or “Suppose a user asks you to do this thing and then you do this—would that be aligned behavior?” You can pose it in lots of ways, and clearly they seem to be able to answer this question: “Oh, yeah, no, that was not the desired behavior.” But they still do it.

I think we’ve got to the point where one hope might have been, “Oh, the problem was just that the system was dumb. Once it understands what we want, then you should be able to plug that in somehow to get it to do what we want.” But it’s somewhat interesting that we’re seeing it’s not trivial to do that, even when there is a commercial incentive to do that. That doesn’t mean that we won’t.

I think it’s quite plausible that we see the obvious reward hacking being fixed pretty thoroughly pretty soon. People tend to say, “Oh, yeah, yeah, we just haven’t put the best—the really good people—on it yet. It’ll get fixed soon. Once we actually focus on it, it’ll be fine.” I’m not sure, but there’s at least some evidence that it’s not trivial to connect the fact that the model knows this is not what you want with its not actually doing that.

I mean, I think you said it was much more common on RE-Bench than HCAST, and you also tried to remediate, right? You can say, “Please solve this the intended way.” Some people prompt language models with something like, “We’re solving cancer here. This is really, really important that you do it the right way.” Some of those remediation prompts actually seem to make it more likely that the model would reward-hack.

It’s a little bit like saying, “Don’t press this red button,” and then it will press the red button. What can we actually do meaningfully to stop this happening?

Yeah, empirically, this seems to happen more on tasks that are more clearly in the RL distribution rather than the chat distribution, on things that have a clear number. The most reward-hacky situations are when the agent thinks it’s going to fail otherwise.

Obvious short-term mitigations are to check your RL environments more carefully and read more of your trajectories. You can train these models to read what the models are doing more carefully and not reward them for doing things that are obvious hacks.

I think the concern there is that, if you have some detector for reward hacking and you train against it, you may just overfit to the detector and make your reward hacks more subtle. Or you train the model to persuade the detector to approve the thing. It’s sort of scary to be in a regime of training against your best ways to know if your problem is happening, because maybe you just get the silent problem.

For current model capabilities, sometimes it’s kind of expensive to have a human check them, but most of the time it’s not beyond human capabilities. The harder version of the problem is when we’re hoping that capabilities will generalize beyond things that we can evaluate, both because of generalization and because we can train on problems even if we wouldn’t know how to solve them or how to look at part of a solution and understand whether it was doing what we wanted.

But we can check the number that comes out at the end, and that’s a signal that we can use to improve capabilities. We’re going to be in this regime where you can be superhuman at making numbers go up, but it’s unclear whether or not you’re actually getting what you wanted.

Yeah, exactly. And I suppose there’s a monitoring problem as well, right? We could, in principle, look at the agent transcripts, and I know you’ve spoken a lot about neuralese, as you call it. Saurabh Karamcheti had a paper out called Chain of Thoughtlessness, which is basically that there’s very little, and sometimes no, relationship between the chain of thought and what the model is doing.

Melanie Mitchell discovered similar things: on the ARC Challenge, even though it will get the right answer, if you look at the chain of thought, the reasoning is completely off. With deception and scheming, and with situational awareness, it might actually know that it's being monitored, and it might tell you what you want to see.

I suppose you could argue that maybe the thinking is almost like the press secretary. It's not the orchestrator. It's confabulating through a different path in the network, and it's not necessarily related to the mechanism of thinking and deriving the result. I don't know if you'd agree with that.

Yeah, I think it's clearly related in some cases. People use a few different words for this, like the faithfulness of how accurately it represents the computational process happening inside the model. We think somewhat more about monitorability, which is: Is it possible for the model to do some kind of computation without you being able to detect that in the chain of thought?

For some things, clearly the model can just do it in a single forward pass. It doesn't need the chain of thought, and it could just have a thin thought about something totally else, and it would be totally unmonitorable. But there is a level of complexity where the model cannot do it in a single forward pass and also, as far as we can tell, cannot do it by reasoning with the tokens in a way that's so different from the natural-language understanding of them that we can't see roughly what it's thinking about.

There's also a sort of nuance here around chain-of-thought controllability. It might be the case that the model doesn't actually need the chain of thought to do the computation, but it isn't able to stop itself from blurting out things that are at least related enough to what it's doing that you can tell, “Oh, it's thinking about this. It's trying this thing that we didn't want it to do.”

Some ways this could go away are general capabilities improvement and being able to do more in a single forward pass, or potentially just doing more RL training such that the chain of thought—the way the model is using the tokens—is too different from how a human would interpret them, so that we don't really know what's going on. Another possibility is moving to recurrent architectures, where you're not going through these discrete tokens; you're just passing vectors around.

Okay, that makes a lot of sense. And just closing the loop on this notion that they are agents: You were saying before that we can adopt an instrumental fiction. Basically, we can say they behave like agents, therefore they are agents—similar to Dennett's intentional stance.

But I suppose the deflationary view is that the models exploit scoring loopholes under optimization pressure. The inflationary view is that they are scheming. I wouldn't call exploiting—I wouldn't call reward hacking scheming.

Ryan Greenblatt

Oh, interesting. I think people usually use scheming to refer to the model doing what it's currently doing in service of some long-term goal and deliberately doing things like appearing aligned or getting a high score in service of eventually accomplishing that goal. You can be reward hacking in an extremely dumb way, like the boat example, where it's just, “This is what RL kind of found,” or, “This is what an RL search found.”

Or you can be reward hacking in a slightly more interesting way, where you actually have the goal of making reward go up, and there's planning and stuff going on about that. But these would all be distinct from scheming.

Yeah, I guess I'm trying to understand the distinction. You're saying there are examples like the boat going around, and that's obviously degenerate behavior. You wouldn't interpret that with an intentional stance; you would just say that's degeneracy.

When the sophistication increases, we might adopt an agential stance and say, “Oh, it's in service of some bigger goal.” But the problem I have is: Is it always just an interpretation? Could we have a mechanistic or strong definition of when something is being an agent?

For the specific question we're discussing, the test is what it actually does in some circumstance where it has the opportunity to achieve this long-run goal. We might not be able to actually observe this, but you can talk about what observations would make it one or the other.

Will this agent, in practice, when it has some opportunity to make the reward go up, do that? Will it only do that? If the agent is more like the RL algorithm, it'll be like, “Oh, it will do that once it's explored it by chance and gotten a reward, and that's been reinforced.” If it's an agent that can reason about the world and plan, it will do that once it learns the facts about the environment that let it infer that.

If we're talking about some long-run goal, it would do it when it actually has the opportunity. If we're talking about takeover or something, it's not going to attempt anything while it's under full human control, but once it's deployed widely enough or has sufficient capabilities to actually succeed in a sort of coup, then it would do that.

That is the thing we're trying to predict. The question is, how can we predict that given the observations we do have? We've never put it in that situation, and we just have this behavior, which is maybe indistinguishable between, “It was a totally nice model doing what we wanted, and it's just going to continue to do what we want in a predictable way,” versus, “Ah, yes, it had this other goal, and it's doing what we want and looking like a nice model because it predicts that that will lead to it getting more power.”

And on Rob's podcast, you said something that was quite surprising to me. You said that AI could autonomously self-improve within as little as 2 years, and maybe even shorter timelines were hard to rule out. Could you walk through the concrete sequence of steps that could lead to that kind of recursive self-improvement?

Sure. Yeah. Maybe I'd put a whole-number percent on this year, but a low whole-number percent or something. Ask me on different days and I give a slightly different number. It seems very unlikely to happen this year, but it's not unlikely enough to rule out.

I think that basically looks like maybe we would see an accelerating trend in time horizon on easily hill-climbable tasks. It turns out that was actually a much more general capability, and there was just a bit of something you needed to do to elicit it on these less hill-climbable tasks. Fundamentally, they're using the same capabilities in a model; it was just what you trained on that was affecting the difference we were seeing.

Then this is leading to—you automate and accelerate a bunch of R&D. There are a lot of low-hanging fruit, even things that we already know you could do and that would improve model performance. It doesn't require new breakthroughs; it's just labor-intensive to do. So you make much better post-training environments and really craft them to teach all the new abilities that you want.

I think you can probably improve compute efficiency a bunch with, again, just applying more labor to making all your kernels more efficient and also doing the right kind of routing between different models or other things like that. There are lots of ways in which how we're using compute is not optimized, so you could potentially get the equivalent of much more compute scaling out of that.

Then another piece would be scaffolding and training the models to use particular scaffolding, and to use memory and retrieval in the right way. It seems kind of obvious that if you really had all the right training data, and you have a transformer, and it can fill its context with different things and take stuff in and out, it can do a pretty good job of something that looks like continual learning or building up understanding.

If you've got a massive context window and quite a lot of bits in there to add things about what you've been learning, and if you really had optimized the training for all of that, maybe you can get that to work pretty well. Maybe some other piece would be, “Oh, yeah, models are kind of superhuman at predicting the results of experiments because they've read so many papers, and because they can predict experiments and synthesize things from different fields.”

And again, maybe it's possible that we're not seeing that good performance here just because we haven't quite elicited the models to do it. It's not a thing that they've seen humans do, but they actually sort of have the capability in there. So maybe you can make much faster progress if you can do a bunch of iteration. You don't actually have to run the experiments. Models are much better at predicting what will and won't work.

And then, when you do run experiments, you can run a bunch more of them because you can optimize the code with your very fast coding models. As you do a few more rounds of this, you get to a point where you train on a bunch more things that are good, high-quality task proxies for what you want, and you get enough generalization to the things that you can't directly train against.

I think intelligence is not capability. I think it's the capability to acquire capabilities. We are in different parts of the phylogenetic tree, I guess, in that respect. But what do you think is the gap in my interpretation?

I'm personally not worried. I don't think the models today are intelligent at all. Obviously, your position is difficult for me to grasp, but I don't know what the difference is. What do you think?

Maybe at least some of it is just this probabilistic thinking about the world, where I'm uncertain about what intelligence is, and I have enough probability on models having it to be thinking about what would happen if that's true. But you clearly think it's more likely than I do, so we could just talk about that difference.

Models have this jagged frontier. There are things that they are much worse at than humans, some kind of generalization and sample efficiency. And there are things that they're much better at, like speed and cost. Maybe you can use these to compensate for the others to some extent.

If you're not good at designing your code nicely, maybe you just have to rewrite it from scratch every time. But maybe that's fine if you're a model and you can output tokens like nobody's business.

It's some combination of thinking that the spikiness is evidence that we should interpret a given level of capabilities as less impressive in terms of reasoning or inference or something, because we know models have so much knowledge. But it is also true that they do have a ton of knowledge, and they will continue having a ton of knowledge about things.

Maybe there's some question about how far you can get by being, in some sense, not very good at sample-efficient learning, but just extremely knowledgeable, and how much you run into things where you now need new knowledge and you can't produce it in some incremental way, or you can't generalize it enough.

Just a quick comment: I think knowledge is the crux. I actually think that intelligence is overrated. François Chollet put a post out saying that intelligence isn't a unified variable. It's measured differently in different domains. You can't meaningfully measure the domains together, and it's not a thing that just keeps getting higher and higher.

It's more like a ball becoming more smooth. As you become more intelligent, the ball becomes more smooth. He thinks that we are quite near the optimum of being a smooth ball.

I don't think we are. I don't think we're very intelligent at all. I think a lot of our creativity is through us being a collective intelligence, and we have deep, grounded understanding and perspectival understanding.

LLMs are a bit of an interesting case because they're like a library. So they know everything. They have the perspective of everyone and no one at the same time.

Experts like yourselves—you can prompt a language model and make a simulacrum agent of Beth, and you can make the agent think like you, and that's very valuable. But you also need all of the different perspectives, and you almost need to create a society of grounded agents creatively exploring things.

When you just have the library on its own and you put it in an agentic harness, you can make it do a specific thing which is well specified.

I think in the AI and automation scenario, I'm definitely imagining that you have a large number of agents because you have all this agent labor. You can do lots of specific, different fine-tunes or different kinds of scaffolding, and accumulate knowledge in some kind of store that all the agents can interact with.

Maybe you're thinking of this as more of a paradigm shift, and I'm thinking of it a bit more as, obviously, if you iterate on the current agent paradigm, you add some more things to your scaffolding. That's not fundamentally that hard or something.

I agree that if you had current models and gave them one system prompt, you then can't plug them into being a call center worker and dealing with all of the edge cases that come up. Maybe there's some difference in how much you think this has improved between GPT-2 and where we are now.

I would say the amount of adapting to new things that are happening that models could do now does seem like it's much higher. They're much better at editing their own scaffolding or reasoning about their embodiment—knowing not to kill your own process or stuff like that. There is a trend of improvement.

I think it's also just having some probability on there that there is an elicitation gap on particular things, and that maybe a lot of taste is basically just being able to predict the results of experiments. You think about all of the things that you would try, and then you can quickly be like, "That wouldn't work for this reason. That wouldn't work for that reason. That wouldn't work for that reason. Oh, actually, someone in some literature in some different field also tried that, and we already know that won't work."

In some sense, models should be quite good at that. It's plausible to me that you see big gains once people figure out how to actually train on that. Maybe you need some amount of expensive-to-gather training data that people haven't bothered to get yet, but you don't need a huge number of data points because you're not instilling this whole new capability.

You're just eliciting: "Okay, actually use your knowledge of all of the papers you've read in all of these different fields to iterate through these. No, these ideas aren't promising; these ones are."

Again, I think this is one of the things that I more think of as being measured a reasonable amount within just doing this 8-hour ML task in a novel domain, with this weird constraint or something. It does seem to me like you have to do some amount of thinking: "Okay, which things are promising to think about? How would I know if this is making progress? How should I allocate my time? I've got some limited time and resources. How should I allocate my time to what's most promising?"

You have to be able to do some of that. Relative to humans, models are doing more of just—well, they're quick to implement things, or they implement them better, or they implement more things and then get to test them or something. But I think it would be surprising if there's none of that.

If you are seeing performance on long, verifiable tasks that are very hard, then in the middle of those tasks, where you don't directly have a signal, you're doing this non-verifiable-task thing of choosing what to spend your time on, choosing what approach to pursue, and deciding whether that was actually working.

In some sense, you could put some metric on it—make $1 billion or something like that. You'd be like, "Oh, this is actually a verifiable task because there's a number at the end," but that can still involve a whole load of things that look more like what you're describing and less like dumb hill climbing.

Folks, I think we've run out of time, but it's been such an honor to have you both on. Maybe, just in closing, could you both say what the single biggest inference is that people out there should be making from the research that you're doing? Thank you both so much for coming on. It's been an honor.

To me, the biggest thing is AI. It might really totally transform the world economically and socially.

And I don't think that it's certain exactly how that'll look, but I think the rate of progress speaks to that.

It is possible both for things to currently be overhyped and exaggerated and less impressive than they look, and for it to be the case that in the future this thing is going to be a big deal and you should be worried about where that's going. These 2 things can coexist, and I think people often have positions that are surprisingly correlated on some axes, like how soon you think AI is coming or how good you think AI is. I'm like, no, these things could all be separate. People can be wrong in different directions simultaneously, or whatever.