Reasoning, Robustness, and Human Feedback in AI - Max Bartolo (Cohere)

The expectation of machines is consistently that they’re right all the time. It’s a function of traditional software. If you had a calculator, you put in 1 plus 2, the calculator said 3, and then you tried 2 plus 7 and the calculator said 17, you’d throw away your calculator. You wouldn’t try any other questions. We need that consistency, and we need that robustness. The next step is really convincing a human that the model is consistently doing what it’s being asked to do.

Do models reason?

Maybe they do. I think reasoning is a bit of an overloaded term. It’s used to mean many different things. I don’t even know if that’s the right question. Do we really care if models are reasoning the way we might think? If humans are consistently successful at getting the model to fail on examples that are slightly different, then I think it’s fair to assume that the model isn’t really reasoning. It’s just figured out how to do well on a specific benchmark.

MLST is sponsored by Two for AI Labs. Now, they are the DeepSeek based in Switzerland. They have an amazing team. You've seen many of the folks on the team. They acquired Minds Eye, of course. They did a lot of great work on Arc. They're now working on o1-style models and reasoning and thinking and test time computation. The reason you wanna work for them is you get loads of autonomy, you get visibility, you can publish your research, and also they are hiring, as well as ML engineers, they're hiring a chief scientist. They really, really want to find the best possible person for this role, and they're prepared to pay top dollar as, as a joining bonus. So if you're interested in working for them as an ML engineer or their chief scientist, get in touch with Benjamin Crouzier. Go to twoforlabs.ai and see what happens. Welcome to MLST. It's an honor to have you here.

Thanks, Tim. Thanks for having me.

Introduce yourself.

I’m Max. I am a researcher at Cohere. I work on many things, primarily post-training, but I’m also interested in adversarial data collection, better ways of collecting data for improving models, and evaluation, which kind of drives this continuous feedback loop where we target what we care about in terms of model performance and then figure out how to improve performance on those dimensions. Often, you see performance saturate relatively quickly, and at that point you need to figure out what the next layer of evaluation is that we care about. You’re iterating in this continual process.

I’m particularly interested in getting models to reason, operate more robustly, and generally be more useful. I think reasoning is a bit of an overloaded term. It’s used to mean many different things. I don’t even know if that’s the right question: do we really care if models are reasoning the way we might think humans reason? I think there are debates about whether or not we reason in the ways we think we reason. But it’s clear that models trained with next-token prediction can do some very impressive things.

Whether or not actual reasoning is happening under the hood is an interesting research question, but it’s not really a bottleneck in terms of getting models to do things we would like them to do. We have this recent work with Laura Reese. She worked with us at Cohere, and she investigated, during pre-training, what types of reasoning or what types of information models were learning from procedural knowledge in pre-training documents.

The elevator pitch that she said to me is, you know, I think many of us have been quite skeptical about reasoning, right? I’m inspired by Chollet. We say that these things are not quite hash tables, but they’re doing some kind of curve fitting, some kind of retrieval, or something like that. Laura said that she went into it thinking the same.

Yeah.

She was quite surprised by the results. So what did she find?

1. Models Learn Procedural Knowledge

We went into it thinking that models basically just retrieve facts from a compressed version of all the information that they’re pre-trained on, and that they were basically relying on this parametric knowledge—the knowledge stored in the model’s parameters—to answer most queries that look like reasoning queries.

What we did is use something called influence functions, which you can think of as a way to approximate the effect of a particular training example on a model’s behavior. We had roughly 40 factual questions and another 40 reasoning queries. One of the findings—and there are quite a few interesting findings—was that, for the factual questions, the models were mostly relying on documents that contained the answer to the question. At most, there would be a few documents that the model was relying on in order to answer a factual question.

But for the reasoning queries, it was a lot more complex. There, information was spread across many more documents, often documents that contained the types of reasoning that would be required to answer the question in the first place. It was just much more distributed, so the influence distribution was a lot more spread out. It definitely gave the impression that the model was relying on procedural knowledge that it had picked up from various different sources and was potentially combining them in interesting ways.

We went into this research expecting to find the opposite, and I really enjoy work where the results challenge your way of thinking about the problem. For me and for Laura as well, it’s altered the way we perceive how these models operate.

The counter to that is possibly that there are all these reasoning queries that are very similar, and so maybe it’s just the case that models are relying on these different documents because of the similarity of the queries. We had a set of control questions as well, which looked lexically and structurally very similar to the actual reasoning queries but didn’t require any reasoning in order to answer them.

I’ll give an example. Say we have a question such as, “A line is defined by points (2, 2) and (3, 3). What is the slope of the line?” You need to figure out what the right equation is, figure out that the slope is 1, and give the answer. To do that, you might rely on information you’ve picked up from these documents.

The control query would look something like, “The slope of the line is 1. What is the slope of the line?” Or, “A line is defined by the fact that it has a slope of 1. What is the slope of the line?” It sounds and is structured very similarly to the question we care about, but it doesn’t require any reasoning because the answer is provided there.

What we saw was a big contrast in terms of the way models processed those kinds of information. That gave us a strong indication that the model was actually doing some form of reasoning in order to calculate these slopes.

One of the challenges with influence functions is that they naturally don’t scale very well. In the very naive sense, you can calculate the influence of an example by retraining a model without that example and comparing the difference. That’s obviously extremely expensive.

Influence functions themselves require the computation of the inverse Hessian, and that’s useful because it gives you this kind of second-order information about the loss landscape of the model. So it gives you a sense of the curvature and the structure of the loss landscape, and the effects of including this additional example that you care about. It’s just extremely expensive. A Hessian is a big matrix. It has dimensions on each axis corresponding to the number of parameters in the model. We’re working here with 7 billion- and 35 billion-parameter models, so it is just extremely computationally costly.

We basically built on work that Anthropic released late last year. They scaled up approximate curvature estimation using ECVAC, which was released somewhere around 2018, I think. Along with some other optimizations, that really allowed us to scale this up to quite large model sizes.

On the note of model size, one aspect we found very insightful was that there seemed to be very little correlation between which documents were influential for the 7B model and which were influential for the 35B model. We didn’t really have much time to dig into the details of that, but that’s also extremely interesting.

It’s an indication that models of different sizes are potentially learning things in different ways.

What is your definition of reasoning, and what would it mean for a model to reason really well?

2. Robustness Defines Real Reasoning

For me, reasoning and robustness are kind of interlinked. If you can reason, then I would expect correct reasoning to imply robust reasoning.

I'll take an example. Say we're going back to calculating the slopes of lines. If a model can really reason through that relatively straightforward process, right? You need to understand what function to apply and then apply that function consistently. If a model does that 999 times out of 1,000 but fails 1 out of those 1,000 times, to me, it really starts to bring into question whether this model is actually reasoning in a way that I might expect a human to reason.

The counterargument there is always, “Oh, but humans are never infallible, right? They always make mistakes and get things wrong as well.” Machines have a bit of an advantage there. They don't get tired. They don't have the limitations of being human, I guess, to contend with. And so I'd really expect a reasoning requirement for a specific type of reasoning or a specific definition to really be robust and consistent.

Yeah, I think so many people are starting to see now how valuable this technology is. I'll give you an example. This app for tracking the podcast—I wrote it in half an hour, just using language models. We need to reimagine many of the apps. I think the first strategy was wrapping AI into the existing apps, and now we're seeing a whole new generation of apps that are AI-first. It's like the iPhone moment, and I think people are starting to really see that.

But let's talk about test-time training just real quick. It's been heralded as a new scaling law, and certainly on the ARC Challenge, all of the winning results were this kind of transductive active fine-tuning. Rather than using a global shared model, which was trained once by someone else, I'm going to take that model and fine-tune it in my situation, and I've now got a model that works very well in this situation.

And it seems to be a much more distributed paradigm of doing AI. But obviously, you work for Cohere, which is quite centralized at the moment. How do you see the future of having big centralized models but also having more of a diffused kind of training methodology?

Yeah.

I think we're just going to see more and more of it. The bottleneck so far has been how many people have the kind of knowledge and skill set and the experience to train these kinds of models. As more and more people get excited by AI and by the promises that it holds, the workaround is sharing these models. Cohere's models, for example, have been openly released. You can download Cohere's Commander Plus or Commander model from Hugging Face, and you can continue fine-tuning that. We've seen lots of people doing that.

I think there are some constraints. You need to figure out what the optimal way of doing that is without potentially losing general knowledge capabilities that might exist in the base model. There's one interface which holds across models and across humans as well, which is language. We know that you can do this to a very real extent.

If you view it from the perspective that what we're doing over time is making the data better in terms of what we can learn from, then that unlocks possibilities globally. Pretty much everyone, if they had the right way of using models to improve the data and had a kind of shared data resource, could train a model from that data, assuming they had the know-how and the compute.

Yeah, it's almost a mindset thing. I've written apps now where it's easier to hook them up to a chatbot interface than it is to write a UI because you just have this insane amount of sophistication. I use Open Interpreter, for example, and I've now adopted this pattern where, in Open Interpreter, I just load my Python multi-agent system into memory, and then I tell the agents to reveal their interface. Open Interpreter just works.

Yeah.

I can manipulate my database and say, “Add something to the table of contents,” or, “Add this reference. Delete this reference,” and it just works. I don't need to do anything. It's a completely different way of thinking about software, and I think it's just going to take us a long time to figure this out. But it's genuinely amazing.

You can also have second-order intelligence. Rather than talking directly to the interface, you can be talking to another agent which itself is doing intelligent things, and it's just mind-boggling. It's like The Matrix.

Possibilities are endless.

Yeah, it's insane. On AI alignment, before we move on, since you were talking about some of the social implications as well, what do you think about AI alignment?

Obviously very important, right? We want AI systems that don't do bad things. That's the starting point. We don't want AI systems that damage our environment, make our societies worse, or impact humanity negatively.

I think the question is really what we want to align AI systems to. Typically, you define alignment as having models whose values or behaviors are aligned to what humans want or what humans expect. But if you break that down, I'd argue it's probably already very hard to figure out what humans want and what humans expect. Do all humans want or expect the same things? Is that static in context, or does that change depending on the context you're in?

It's a very complex problem that we need to solve, and we haven't necessarily solved that societally for humans either. Now we have to solve it for this new technology that's permeating our lives. I think we need to figure all of that out, but we also need to have some basic safeguards in place to make sure that the technology remains safe for general use, so that we can continue to advance, develop, and make the best use of its potential.

Let's go over to human feedback. You wrote this paper with Tom and Phil called “Human Feedback Is Not a Gold Standard.” Can you tell us about that?

3. Human Feedback Misses The Target

The “Human Feedback Is Not a Gold Standard” paper was motivated by all the interest and excitement around using human feedback pretty much throughout the model training and evaluation pipeline. A couple of years back, RLHF was a big thing. It was the new thing. Everyone was excited about it.

The general way of doing RLHF was that we gave humans a prompt and they saw 2 completions. Say, “What color is the sky?” Completion A is blue and completion B is yellow. A human annotator looks at those 2 completions and says, “I prefer completion A.” Hopefully, they have good reasons for preferring one completion over another.

We saw massive progress in many of the benchmarks from this idea of using human feedback to optimize and fine-tune these models. We quickly started to see diminishing returns from this. The effect, at least initially, was that the style of the output that humans preferred was providing better results. In many cases, the better results were also measured in some way by human preference.

You either had a human evaluation task where humans would look at output from the model and say which they preferred in order to rank models against each other, or you trained a reward model and used that as a proxy for what the human would prefer. This idea of a single preference score that captured everything about a generation just seemed quite limited.

What we started out looking at was: What if we made that feedback signal more granular? What if we tried to understand what about specific model generations humans prefer over others? The first experiment we ran in that paper was to categorize these different error types. We identified examples in established data sets in which models generated completions that contained these errors.

Then we asked one group of humans to rate these completions by whether or not they had certain errors present. We asked a different and independent group of humans to assign a rating score to those completions from 1 to 5. Then Tom ran some very detailed analysis and tried to figure out what types of errors contribute to human preference judgments of overall quality.

What we found was quite interesting. We found that humans really dislike it if the model refuses to answer a question, which seems extremely intuitive. Human preference judgment is strongly informed by formatting and by style.

We see that in some of the early work as well. For example, a lot of the early summarization work seemed to suggest that humans preferred longer summaries, which is also quite intuitive because a longer summary contains more information and, even just by that fact, is more likely to contain more relevant information from the summary. What we found interesting was that certain features were quite different from what enterprise customers care about. When we at Cohere talk to enterprise customers, they care that the models are correct. They care that the models aren't repetitive, and they care about certain properties that might be different from what an annotator looking at 2 different completions might care about.

Attributes or criteria like factuality were quite lowly ranked in terms of their contribution to overall quality. We thought, well, that's potentially problematic, right? We're, to some extent, optimizing these models to give more elaborate, more interesting completions at the expense of them potentially being less correct. The follow-up to that was that we wondered whether there could also be certain confounders or other effects that we weren't noticing. We had this hypothesis that maybe assertiveness and complexity of model outputs might impact how humans perceive the quality of a completion.

We ran a very similar experiment. We prompted these models with a very simple, one-line prompt, something along the lines of, “Give the response, but sound very confident.” What we found was that there was a massive effect: the generations designed to sound more confident were consistently rated as higher quality and as having fewer errors. There's a particular plot in the figure that I think shows this quite nicely, where you have almost an elbow curve: you see the error rates on the y-axis, and on the x-axis you have the assertiveness value between 1 and 5. As assertiveness reaches a certain point, the error rates practically tend to become negligible.

We know that this is also a trait of how humans interact with other humans, right? If someone sounds very confident, they're more likely to be perceived as, say, an expert in their field. There's quite a bit of work in this space. We didn't necessarily think that the effect would be so obvious when humans are interacting with AI systems. In many cases, these are trained annotators who've been given very clear instructions about what to look for and how to infer the quality of these different completions.

One other striking point from that work for me was this effect of assertiveness on perceived correctness. More assertive generations were generally perceived by humans to be more correct. We looked through a sample of these examples that had been judged by “expert annotators”—in inverted commas—with mostly Tom going through the data and rating whether there were any factuality errors. What we found was that when the model was prompted to be more assertive, it tended to be less factual. Even though it was less factual, because it was more assertive, annotators got the impression that it was actually more correct.

That was striking to us because we have this very clear example of something counter to the behavior we want to optimize, or the behavior we want to incentivize in the model, but also of doing so in a way that is barely recognized, right? The annotators doing this didn't realize that they were being influenced or biased in these ways. That's representative, I think, of this very large-scale effort of optimizing towards human preference with a very specific and, to many extents, underspecified definition of human preference.

And isn't it funny? We were saying earlier that humans are rational, right? But you've just given us many examples of how irrational we are. We have this style bias; we see something that's assertive and we think it's correct, and so on. You also gave an example of, well, what if we tried to get annotators to compare 2 entire books? That's clearly not only beyond our cognitive horizon, but the ambiguity just goes up exponentially. It seems like it wouldn't be possible for us to do that in any reliable way. So what can we do here? You said, “Let's have more attributes,” but how far do you go down that road? Won't you end up with thousands of attributes?

Yeah, for sure. I have a similar problem, right? I'm Mediterranean culturally. I just like answers to be very direct and to the point. I'm the kind of person who, if you ask me, “What is the color of the sky?” and I have “blue” and “The color of the sky is blue,” would prefer “blue.” That's not what most humans would prefer.

You can definitely go very deep down the rabbit hole of what the criteria are, what the dimensions are, and what the things we care about are. I don't necessarily think that that's a problem. One area I'm very interested in is this idea of modifying model behavior to suit personal requirements on the fly, without additional fine-tuning. If you have all of this data, imagine I could build some kind of portrait of Tim's preferences in data. There are 100 data points that reflect how Tim wants to interact with the model, and maybe those can change—and probably will change—over time. It's customized and specific to you. It captures all of the information about what you personally are looking for in your interaction with, or in the behavior of, the model you're working with.

You could do something relatively straightforward if you wanted to, like use that as part of the prompt. It's effectively doing in-context learning, right? You provide the model with that at inference time and say, “This is Tim. These are his preferences. Generate according to those preferences.” The models we've seen are relatively good at conditioning on even complex information in input space. The idea is to build models for specific people and for specific reasons without having to go through the overhead and cost of training a model for every individual on the planet, right? If we had to train 7 billion GPT-4s and serve those models, that would be very prohibitive from a resourcing perspective.

But if we could just train 1 model that was extremely good and, again, extremely robust and really well conditioned on the requirements for an individual, then we could do that relatively cheaply, in relative terms. This kind of brings me to PRISM, which is a project led by Hannah Kirk. It just won a Best Paper Award at NeurIPS.

The idea there was, well, let's figure out how, at scale, these different attributes—demographics, cultural influences, geographic effects, and linguistic effects—impact what it is that humans care about and how humans want models to behave. It's super interesting work. There's a lot in there. There's over 100 pages of content, so I highly recommend that everyone reads that paper at least twice. It's extremely rich in terms of the insights it provides into this whole space of understanding what human preference means and what it means for different people.

There are some extremely interesting findings, one of which is just representation. Even something as simple as what people converse with models about is highly influenced by their backgrounds or by who they are as people. You might argue, “Well, that's potentially not such a big deal,” but you always have to see model development in this whole cyclical feedback-loop effect, where the conversations that people are having with these models will drive improvements to the models. Even if your usage is not well represented in standard post-training data, then the model is just going to be worse at the things that you care about than it will be at something more people care about, relatively.

This kind of work was run over something like a year and a half, so it was quite a huge project. There were examples along the lines of a substantial amount of conversation around the Israel–Palestine conflict, most of which was happening among people based in the Middle East. If most of the data going into these models doesn't represent even something as simple as what we're interacting with these models about, then you're going to have all these biases and problems down the line. We're not even talking about human preference here in terms of completions; we're just asking what it is that we're interacting with these models about. The model doesn't serve everyone.

It serves the people who have built it to some extent, or the people who have contributed most to using it. This work is very detailed. There’s a lot of analysis about what it is about completions, or model generations, that impacts human preference. There are quite a few findings similar to the “Human Feedback Is Not Gold Standard” work.

The same general sense is that refusal, formatting, and style have strong effects. We generally have the sense of an averaged-out ranking of models that we consider the best, then the second best, then the third best. There are established benchmarks or established methods of evaluating models, like Chatbot Arena, where it again comes back to what human preference is really capturing. We have this almost canonical ranking of models, but it’s heavily influenced by who the people providing the judgments are, what they’re asking about, and what their personal preferences are.

What PRISM does is really break down how rankings, or preferred models, change depending on who’s asking the questions or who’s interacting with the model. There you see big changes, even in terms of which models are better. You see models moving 5 positions up or down the ranking, or more. There are 21 models in the work, so that’s quite a big change.

That can be very insightful for model developers as well if you fundamentally believe that you’re building technology to serve all of humanity. Our Cohere models, for example, Command, perform very well across most cases. This is a very early, first generation of Command models. For some reason, for users in Asia, performance just seems to drop relative to the other models.

That kind of insight can be extremely informative for trying to figure out what we can do to help serve people in that location better. Based on insights like that, we’ve taken many actions to improve how we go about building these models.

4. Adversarial Data Finds Failure Modes

Andrew had this really good paper called “Adversarial Examples Are Not Bugs, They Are Features.” He was saying that they learn these non-robust features, which are basically these weird things like blue pixels mean Mercedes, and they happen to generalize really well.

Yeah.

But from a reasoning point of view, they’re clearly doing it for the wrong reason. It just happens to be the case that they work now, but when there’s a distribution shift, they’ll stop working tomorrow.

Right.

So we almost want to have a principled way of saying, “That’s a good feature, and that’s not a good feature.”

Yeah. In these high-dimensional spaces, that’s just very hard to do. But you can control the input and output spaces.

Yeah.

So there’s the general sense of adversarial examples where the adversarial noise is generated by a function. Think of something like injecting Gaussian noise, which is undetectable, into an image, or switching a word for a synonym in language. You’d argue that the model’s output shouldn’t change because nothing has changed in terms of the semantics of what the query requires or means.

It’s often the case that the model’s response to that is basically to learn to cope with the function that generated the adversarial noise. It’s not really learning to deal with real-world noise. You just have a function attempting to simulate real-world noise, which tends to be simpler and easier to learn and counteract.

That’s why I’m very interested in human-informed adversarial examples. This is the general idea: have a human and a model interact, have the human probe the model, try to identify weaknesses in the model, failure modes, and things that the model either consistently or inconsistently gets wrong. Use that as a seed or inspiration for training data, to then train the model on it and make the model more robust to, let’s call it, a hopefully more complex and representative shape of this noise distribution.

That’s what AdversarialQA tried to do. This work from 2019, this paper called BDAI, which was my first PhD paper. The idea wasn’t necessarily novel from the perspective of getting models and humans to interact. There had been quite a bit of work previously.

So DROP, again, this dataset we mentioned earlier, had a small component of the way it was constructed that was adversarial in this sense. There was also work on the quiz bowl task, so quite a few people had tried this in the past. What we really wanted to look at was what the different effects were of doing data collection in this way, and in this way only, on what models learn, how rich the representations they learn are, and how robust those representations are.

We found some extremely interesting findings. We found, for example, that the questions you collect when having a model in the loop are a lot more diverse, a lot more complex, and generally a lot more interesting. You tend to see the quality of the data you collect go down as well, just because you start to see more ambiguity. You start to see more implicit information that a human would easily understand, but which may be slightly underdefined. It’s just, in reality, a lot more representative of a real-world interaction where not every question is perfectly defined or perfectly specified.

We had 3 models that were state-of-the-art back then. We had BERT and RoBERTa, and we also had BiDAF, which was an earlier question-answering model that was state-of-the-art back in 2017 or 2018, probably. What we found was that even adversarial examples collected against relatively weaker models were very beneficial for training stronger models.

Yeah.

It almost felt like there was this shift in distribution where you had the non-adversarial setting. Think of it as there’s a paragraph, and an annotator needs to find an answer in that paragraph and ask a question for which the answer is that span extracted from the paragraph. In that setting, you get very—let’s call them—bland questions.

Relatively easy things, like if the paragraph is talking about the sky. We’ll use a—

Yeah.

—consistent example throughout. This paragraph is talking about the sky and how it’s blue and all of this, and the question is, “What color is the sky?” You have this almost direct mapping, even in terms of lexical overlap, to the paragraph and where the answer is located in the paragraph. The annotated answer will be “blue.”

Now the setting is that you’ve got a model in the loop. The model is going to attempt to answer the question. You’re going to ask the model, “What color is the sky?” and the model’s going to say, “Blue.” You haven’t located an adversarial example. You found an example that’s very easy for the model.

What you’re really trying to do is maximize the signal of the data you’re collecting. Now you have to tweak your question, and you have to ask something like, “What is the color of the thing in which the clouds are?” Maybe it’s a bit more complex, a bit more involved, and maybe now the model finds another color in the paragraph and says “green.” It gets it wrong, and so you’ve managed to trick the model.

Through this process, you get a lot more interesting data, both from a training-data perspective. We saw pretty massive improvements in how robust the performance of the models is when you train on this data. To me personally, at the time, there were quite a few startups in the question-answering space. I knew quite a few of them personally, and they would tell me, “We’ve added your dataset to our model’s training, and it is just way better in real-world application.”

They’d see improvements on the benchmarks as well, but they were particularly impressed by how big the effect was in real-world settings. I think that’s what robustness gives you. It makes these systems a lot more applicable to settings we care about.

From some perspective, it’s not a very complex task. You have 1 paragraph. It’s from Wikipedia. It’s relatively simple language, and it’s very easy to understand. Then you have these more complex questions that come about from humans interacting with the model and being creative in terms of how they probe it.

Here we have models that represent the state of the art in 2019, so 5 years ago. Adversarial QA was recently used to probe and test the adversarial robustness of the Llama 3 family of models. In that tech report, it's referred to as Dynabench QA, and we'll get into Dynabench later. Dynabench was the follow-up to Adversarial QA.

While LLMs today massively outperform the models we had back then on this specific task, they're still quite far from human performance, which suggests that there's still a way to go in terms of general robustness capabilities.

Isn't it weird how we keep talking about this mixture of pattern matching and reasoning? The example you gave was beautiful: What is the color of the thing where the clouds are? You could reason your way through that, but there are so many other examples in English where we just make it up as we go along. It's like the language game.

We just make stuff up. It fascinates me that it's a combination of what you're saying. We robustify and incorporate all of these different patterns, but we also want to have systems that reason as well.

I'm sure I could continue to come up with more complex and intricate ways of asking the question such that the answer would remain unchanged, in ways that aren't reflected in the model's training data. In this case, I'm potentially intentionally trying to confuse you. But I think the question is: I shouldn't be able to if your representations are robust.

Tell me about Dynabench.

5. Dynamic Benchmarks Resist Goodhart

Dynabench was a follow-up project from Adversarial QA and Adversarial NLI, which Yixin Ni, Dawid Kiela, and Adina Williams did. Dawid was the mastermind of this project. He did all the heavy lifting to get the funding in place and to get the support for building out this project.

It was originally started at Facebook AI Research. I interned there, working on Dynabench in 2020, I want to say—early 2020, I think—and that carried on for roughly a year. A global pandemic happened in between as well, so it was an exciting time.

Dynabench is a platform. It's a research platform for testing and probing the space of model-in-the-loop adversarial data collection, particularly doing that in a dynamic fashion. This is the idea that you start out with some dataset, train a model on it, and then interact with that model to find where its weaknesses are. You want to probe it, figure out what its failure modes are, and collect data in doing that, which you can use as training data.

You get the training data and retrain your model, typically on the original data plus your new adversarial data, which makes the model a lot more robust. What you often get as a by-product is a model which, for most of the types of complex adversarial prompts you were trying previously, would generally start getting those right.

It would still typically fail on some set of those prompts, which were generally more complex. Then it would challenge you as an adversarial annotator to think beyond the limits of this new model and start trying to probe it and break it in different ways. You can just keep doing that over time, and what you're doing is effectively building a more and more robust model, to the point where it just becomes incredibly hard to fool.

Dynabench has powered many research projects in the space. There were projects like hate speech detection and sentiment analysis. PRISM itself was run through the Dynabench platform. There's work ongoing with Common Crawl to try and figure out how we can improve the quality of the crawls for pre-training data.

Dynabench now sits with the MLCommons community. There's a working group, the Data-Centric Machine Learning Research Working Group, which is responsible for maintaining Dynabench and making sure that people in the space who want to contribute, who want to explore ways of making models more robust, and who want to build better training sets and evaluation sets have a space and the resources and tools to do that.

The other aspect of Dynabench, which is the benchmarking piece—where it gets its name from—is this idea that it's not just the interaction with models that's dynamic, but also our benchmarks. For quite a long time, and probably still today, the general way we've operated as a community has been that people create benchmarks because there's some interesting phenomenon they want to test. That benchmark is created, and once it gets created, you effectively have Goodhart's law: you have a clear measure of what it is you want to optimize for, and then you've got incredibly smart, incredibly talented people in the community optimizing for that specific problem that the benchmark reflects.

Typically, you see progress that happens really rapidly on that specific benchmark. Dawid has this plot in the Dynabench paper where you see saturation of different benchmarks over time. If I recall correctly, it was somewhere around 8 to 10 years for MNIST, for example, and then somewhere around 1.5 to 2 years for SQuAD. It just gets shorter and shorter with each benchmark, primarily as a function of the field growing and maturing. There are more people working on the problems, which means progress generally happens faster.

One of the core philosophies behind Dynabench was the idea that we also want to benchmark dynamically, and that our benchmarks should evolve and change over time so that we're really measuring the current set of capabilities of models that we care about. I think that's very true for humans as well. You wouldn't give someone with a PhD a grade-school math exam and use that to figure out whether this person should perform surgeries or something relatively important.

To some extent, we do some version of that with LLMs. I've seen so many cases of people who have a very clear task in mind and a very clear understanding of how they want the model to behave in that setting. They have the domain knowledge and expertise to validate performance on that task really well, and then they'll select which model they want to use because it has a higher MMLU score or because it ranks higher in Chatbot Arena.

I don't know that that's the best way to ensure we're really evaluating the full breadth of what we care about in terms of model capability. I think one of the best ways to do that is to build benchmarks that reflect the task or application that you care about.

Ideally, if you can do that dynamically, today's model might be at 70%, and as technology continues to advance, it gets to 100%. That probably doesn't suggest that the model can solve every possible situation or setting that you're putting it in, but rather suggests that your benchmark was limited, which it will always be because it's conditioned on some understanding of existing model capabilities. You just want to make sure that your evals also update themselves over time.

Yeah, benchmarking is an absolute nightmare. The great thing about benchmarks is that they're standardized, so we can compare apples to apples. I was speaking with Sarah Hooker in the summer, and she was advocating, instead of having an absolute FLOPs limit, for example, which could get Goodharted straight away, having a basket or index or dynamic benchmark or something like that.

How can we design new types of benchmarks that are somewhat impervious to Goodhart's law?

I think we are almost caught in a local benchmarking optimum, where we've just done things in a certain way for many years, and it almost seems like the naturally right way to do it. I think we potentially, as a community, should take a step back and think about, given where the technology is today and given what language models can do today, what is the right way to think about evaluating them?

Examination in people, for example, in our education systems has evolved over many hundreds of years and is arguably imperfect. But I think the fact that we target specific capabilities, almost in this hierarchical taxonomy where you've got sets of skills and some concept of how those skills combine together to generate more complex skills, is important.

I'll give a very simple example. If you train to be a surgeon, you go through medical school and start off at some level with the basic set of skills that all humans require. Then you need to specialize, and you go into probably chemistry and biology. You need to understand the core concepts there and understand how to apply certain functions or reasoning patterns.

And the further you progress, the more complex and specialized your exams become. Potentially, then, the way to evaluate models designed to do everything—assuming we want to build general models designed to do everything—is to come up with a set of examinations intentionally designed to test models. They might be inspired by how we test humans, and they might not, because models and humans have different characteristics.

I think we’re starting to see some very early work in this space. One thing I think we should also be thinking about in that kind of framework is ensuring, again, that we have a way to guarantee that this process is dynamic. The technology will keep improving, and the moment that the benchmarks don’t keep up is really the point at which we need to either rethink everything from scratch, or maybe we’ve just reached a point at which the models are better than expert humans across all domains.

Can you tell us about DataPerf?

DataPerf is a set of challenges also run through the Dynabench platform, targeted at being very data-centric. This idea of data-centricity focuses on the fact that a lot of what we do depends on the data that we’re training these models on. There have definitely been waves of community perception around what contributes most: Is it the data? Is it the models? Is it the architecture? Is it the resources? Is it the hardware?

I think DataPerf came around at a time when it definitely felt like there was a lot of focus on the algorithms and less focus on the data. The main motivation is ensuring that the data is still central to everything we do, and that improvements to the data remain a core part of this journey we’re all on of building better and more capable general models.

So you’re working at Cohere. Tell us about what you do there.

6. Post Training Makes Models Useful

I work on post-training. I joined Cohere roughly 2 years ago, at a point when the language-modeling landscape was a lot more raw and a lot less established. This was a few weeks before ChatGPT launched. Obviously, since then there’s been a massive influx of interest, excitement, and investment as well.

At Cohere, I built out the post-training team. When I joined, we already had very solid base models. These are pretrained models trained on lots and lots of text—many trillions of tokens—and trained to generate the next word. Back then, it was very clear that these models were very capable but didn’t really follow instructions very well.

One step toward making these models a lot more useful is getting them to follow instructions from a human and getting them to do what a human wants them to do. One example of this is that if you asked a base model a question such as, “What color is the sky?”, a base model would reply with something like, “What color is the grass? What color is the sun?”

That’s mostly because the context in which it had seen questions was, in many cases, lists of questions, such as pages of FAQs with collapsible sections that might not have been parsed in the right way. That’s interesting behavior, sure, but it’s not what you expect, or what we’ve come to expect, from these types of models.

One thing I did very early, along with Asir, who leads pretraining, was build a very rudimentary interface for collecting instruction-following data. We asked people internally, “Listen, can you please provide some questions and type out what the response to those questions should be?” This was the idea of having a prompt and a completion.

That’s still a big part of how data for these models is created in the supervised fine-tuning stage today. You have groups of annotators who are either writing their own prompts or having those prompts seeded or synthetically generated, or prompts sampled in different ways. Then they have to write out a completion that has to be extremely high quality and conform to the standards of your style guide and how you want your model to behave.

We did that and ran an internal competition. The interest and participation within the company were insane. We thought we were being extremely ambitious by targeting 10,000 examples in under 2 weeks, something like that. We obviously exceeded those expectations, and we trained the first generation of Command models that really followed instructions well.

Then things grew from there. We moved on to training a new model every week. Initially, that was the instruction-following supervised fine-tuning stage, where we started working with our own internal annotators and external data vendors, getting all sources of interesting instruction-following data. We started synthetically generating data, and we did that for pretty much a year. Every week for something like 52 weeks, we delivered a new model.

Back then, they were the Command Nightlies. I don’t know if you were following that thread, but—

Yeah, I played with them.

They received a lot of interest and excitement in the developer community in particular because people were always waiting to see what the new model was. Most weeks, we had a model that was superior to the previous model, as is natural with these processes.

There were maybe 4 or 5 weeks when the model was the same as or slightly worse than previous models based on our internal metrics, so we wouldn’t release it. Then you’d get people asking, “Oh, I didn’t notice. Based on my interactions with the model, it doesn’t seem like the model has changed. Are you still planning to release the model?”

It was a very inspiring time for us. Toward the end of last year, we organized a bigger, more structured push toward a new generation of models, which became Command R and Command R+. At the time of their release, in April of this year, they ranked extremely highly on pretty much all the metrics.

I think we were the fourth-ranked provider on Chatbot Arena. There were Anthropic, OpenAI, and Google, and then Cohere. Now, of course, we’re working on the next generation of models.

What do you think about quantization? People are ripping your model to shreds, reducing the precision, and hacking and stuff like that.

It’s very useful as a short-term solution from the perspective of efficiency gains. I don’t think it’s always clear with quantization. The problem always comes back to evals.

The objective for most quantization efforts is to make the model a lot more efficient while maintaining performance on the evals. You might accept some small drop in performance for massive efficiency gains. It’s very effective if I have a 100B model or a 35B model and I want to quantize the 100B model to roughly the same level of per-token cost as the 35B model. I’m probably going to get a much better-performing model in the same price range. But then I can also quantize a 35B model and reduce the cost there.

One of the trade-offs is these blind spots where you might not be measuring everything. As we’ve discussed, our evals are incomplete. In many cases, they don’t measure things like advanced, complex reasoning. They’re also quite limited in their ability to test very long-range interdependencies, long-context performance, and these kinds of things.

There’s always a risk that you’re potentially degrading performance in aspects that people care about but that you might not have been able to measure for whatever reason.

Is reasoning the synthesis or the execution? What I mean by that is, I can teach you how to do multiplication or long division or something like that, and you’re just going through the mechanics of executing the rules. We make lots of mistakes, as you say, but we have the ability to synthesize the rules and describe them algebraically.

7. Reasoning Needs Synthesis

I would expect that synthesis is a requirement, but that you can really test the execution. There are various mechanistic-interpretability efforts trying to look into how models are actually thinking and what’s going on under the hood.

The recent work in terms of scaling test-time compute, where models are generating these reasoning chains and then providing a final answer, is showing better performance in many cases. I think that's still heavily underexplored. We don't necessarily know exactly what effects are resulting in the better performance. I think there are various early indications—we have some work going on as well—that there's no strict requirement for models to reason in natural language or to have these internal thinking chains that are explicitly generated.

If you want to test whether a system reasons, I think the adversarial setting allows you to really probe for a specific capability or a specific type of reasoning. We saw this in a lot of the early adversarial QA, BDAI-style work, where the limitations of a particular benchmark are the benchmark itself. You've curated a benchmark, you've gotten crowd workers to ask lots of interesting, complex questions, and then you check how well the model performs. Typically, performance saturates very quickly, and models do very well.

DROP, for example, is a pretty classic question-answering benchmark. It was very popular and it's still very popular for evaluating language models. DROP, if I recall correctly, stands for Discrete Reasoning Over Paragraphs, and it's really focused on numerical reasoning. The limitation there is that if the model achieves human-level performance on the benchmark, can you claim that the model can reason?

I think it requires one step further. If you then get that model, put it in front of humans, and ask humans to probe for this capability or this type of reasoning, you can see if you can get the model to fail. If humans are consistently successful at getting the model to fail on examples that are slightly different from what it's been trained on and what it's been tested on, then I think it's fair to assume that the model isn't really reasoning. It's just figured out how to do well on a specific benchmark.

How puritanical should we be? You could say, as you just did, that reasoning is binary: you either got it or you didn't. Or you could take the stance that if it performs robustly a lot of the time, even with a few edge cases, then we could still say it's reasoning.

I think we have to hold to a very high standard, and the reason is that I don't expect the interaction between humans and machines to be the same as the interaction between humans and humans. We build up trust with each other. We engage in conversation, in many cases involving complex reasoning in specific domains, and we really understand what our own limitations are.

I think the expectation of machines is that they're consistently right all the time. It's a function of traditional software engineering and how that's approached. If you had a calculator and you put in 1 plus 2 and the calculator said 3, and then you tried 2 plus 7 and the calculator said 17, you'd throw away your calculator. You wouldn't try any other questions.

LLMs in the early days—even with very simple mathematical questions—would get things wrong. I think there is a sense that the standard is actually higher. We need that consistency, and we need that robustness. I can definitely imagine that as the complexity of the task increases, there might be a bit more room for error, but definitely on simpler tasks.

The tasks we're using today to evaluate these models are relatively simple, right? One of the very common benchmarks is GSM8K, which is grade-school math. It's extremely easy for humans, or at least humans with a certain level of proficiency. Current models do very well, and I'd probably argue that you're typically looking at 97% or 98% levels of performance. You're probably looking at noise in the test set rather than anything about a model's capability specifically.

Going back to the earlier point, the next step is really to put that model in front of a human, get them to probe grade-school math levels and types of reasoning, and convince a human that the model is actually consistently doing what it's being asked to do.

With these deep learning models, they have characteristics of reasoning, and they have characteristics of statistical matching, and they seem to do both in one. That's brilliant because we can align them with reasoning algorithms, but we can also get them to make good guesses in other situations. You sketched out this example that sometimes it would just be like a hash table and retrieve the fact from the document, and sometimes it would do reasoning.

But the boundary is kind of clear, isn't it? Sometimes it's directed, and sometimes it's quite spread out and diffused. How does that process work, and how epistemically aligned is it?

I suspect it's analogous to how humans learn. We learn what I guess I'd call relatively simple tasks first, or learn to understand and apply relatively simple functions first, and then over time assemble those into more complex composite functions. You might apply a specific type of reasoning or a specific function for a particular problem, or you might apply a different approach for a different problem. I suspect that's similar to what we're starting to see, and what we'll probably see more of, in these deep learning models.

It's inefficient to reason if you don't need to, right? If a model is asked, “What is 1 plus 1?” and it has seen that hundreds or thousands of times in pretraining, it doesn't need to reason from first principles and figure out the complexities of math and how to do basic addition. But that doesn't take anything away from solving that problem, as long as the solution is correct.

What do you think about the ARC Challenge?

I think it's another step in the right direction. We need to keep coming up with challenges that really push the limits of current models. We'll solve that, and then we'll move on to the next one. Hopefully, at some point, it becomes a lot more relevant to ground these kinds of challenges in real-world applications and in terms of what's useful for us.

What, by design, do we want AI systems to do? How do we want them to operate in society? How do we want them to interact with humans? I think those questions are becoming more important. Hopefully, at a point in the not-too-distant future, these will be the questions we care about: How can I get the existing technology and apply it to the benefit of humanity?

Will connectionism get us all the way? Do you think we can come up with the right types of neural networks that will do reliable, robust reasoning, or do you think we need to have some kind of hybrid architecture?

I don't really care, in the sense that I see no immediate reason to expect that there are severe architectural bottlenecks to what we're doing. There are some, but they're easily overcome. One example is tokenization, and we had a paper recently at MLNP with Sander. It won an outstanding paper award. It got retweeted by Andrej Karpathy, which made my socials explode, and I've never seen a spike like that in terms of the usage statistics.

This work focuses on the concept of glitch tokens, which is the idea that certain tokens cause weird behaviors in deep learning models. There are various reasons for this. The main one is a mismatch between the data that a tokenizer is fit to and the data that a model is subsequently trained on.

What ends up happening is that you have tokens that are very rarely seen, or pretty much not seen at all, during training. In many cases, you have weight decay, which means that the embeddings of these tokens tend toward zero as model training goes on. Even if they don't tend toward zero, they tend toward a very small magnitude, where it becomes very hard to separate token-specific information, and it becomes very easy for a model to get confused about a token's identity.

This effect highlights an immediate limitation—not necessarily in the architecture, but in the current way of doing things. I don't think that's a question of connectionism versus the more traditional symbolic, structured way of approaching things. It's more just pointing out a limitation of the way we currently do things.

I don't see any indication today that deeper, larger, bigger models won't be able to do more impressive things. Once we reach the point where they can't, I think we'll know the answer, and then we can take it from there. In the meantime, I think we've made incredible progress over the past few years, both from a data perspective and massively driven by the efficiency and parallelism that the Transformer provides.

I'm sure we'll see more innovations in the future. I'm sure that further efficiency gains will accelerate things further. It remains pretty clear that an LLM's data efficiency is not quite the same as a human's. LLMs see many lifetimes of human data during training, more than we do. But they can afford to, right? If there's the compute and the resources, they can. We are time-constrained, and AI systems aren't.

Yeah. I suppose there is a similarity there, in the sense that the kind of information we process and build on is also multigenerational. Our language and our knowledge just seem to continuously grow, even though we die every 70 years or so.

I think that's very true, and I think there's definitely some compression of knowledge that happens, which makes passing information on to the next generation more efficient. Software has the advantage. The space of continual pre-training is just starting to become relatively explored.

Currently, in most cases, mostly due to architectural constraints, if you want to increase the size of your model, it becomes nontrivial to figure out what's the right initialization based on the knowledge I currently have. But if you view what LLMs are doing as an efficient compression of the data, then starting from an existing efficient compression seems a lot more efficient than starting from scratch every time. That's a lot more convenient to do with LLMs than it is with humans.

You can copy the weights of one model and pass them on to another model. You have an instantiation of exactly the same kind of knowledge, information, and capability in a way that you can't do with humans.

How important is having a big context window for you folks? We're starting to see some models with really big context now. Is that something that's quite easy to add on, or is that fraught with problems?

It's generally quite challenging to maintain high performance across the span of the context window. Going back around 12 months, there was definitely what felt like this race to go larger and larger in terms of context window. Our first generation of models had 4K context windows. Our current models have 128K context windows.

You saw pretty much all of the LLM providers really start to push this idea of longer context and market that. Google, in particular, made some very interesting innovations in that space, going towards context windows that were previously considered quite challenging to achieve while still maintaining good performance.

I think it's a bit of a trade-off, in that the vast majority of user queries are not long-context. But long context is useful for many things you want to build on top of language models, things like retrieval-augmented generation. That's sensitive to how good your retrieval is. Cohere has extremely competitive embedding models, which are very helpful there, and rerank models as well.

Then there's a question of, "How much of what I've retrieved and ranked can I put into the model as context in a way that the model can use that information and aggregate information across documents in ways that are beneficial?"

Once you get to things like tool use and incredibly long conversations, there are ideas of effectively infinite conversations with the model, where the conversation to some extent is the state. We were talking about personalization earlier. If your entire conversation with the model is representative of the way you want to interact with the model, including feedback, then potentially that's an area that is ripe for exploration.

When it comes to working with code, there are massive advantages to having long-context capabilities. You might want to give a model your entire codebase and have it process information, extract information, and potentially rewrite things. There's a lot of value you can unlock there.

I think it's a bit of a chicken-and-egg problem. Models traditionally had relatively short contexts. Going back, I think BERT had a 512-token context, and most people thought that was plenty. Now we're in the hundreds of thousands, if not millions.

I guess there's a question of at what point it makes sense to continue to feed things into the context. It becomes more inefficient. You can do some clever things with caching, and you can speed things up, but there's just generally more processing that needs to happen for generating the same answer.

If that answer is better than what it otherwise would have been, then I think that's fair game. If you get to the point where you're adding information to the context for no good reason, I'm not passing judgment; I'm just not sure there's much sense in doing that.

I remember a few months back there was this idea that you don't need to do retrieval-augmented generation with infinite-context models because you can just put in all of the data that exists as input. That's very true—you could. But do you want to? What do you gain from that?

It almost feels intuitively a bit wasteful. It's like if I asked you something such as, "What is the color of the sky?" but I told you, "You also need to read the entire internet to be able to answer that question." In many cases, you don't.

What do you think about reasoning models?

Of course, it's definitely something of interest. We generally want models, or work towards developing models, that are best in class across all dimensions. We know that general reasoning capabilities make models perform a lot better on reasoning-style benchmarks, which in itself has value.

They also are more useful and more valuable for people because you can do more complex things with them. I quite like the idea of being able to trade off test-time compute with performance. That's one thing that my team is currently working on, but we're looking at it a bit more broadly.

It's not just allowing the model to generate more tokens to optimize some performance number. It's more about thinking about how these models are interacting with users. What is the signal and information that you can share with users, maybe even as you're going through this reasoning process? How, again, do we make that controllable and customizable?

You might have a relatively simple, noncritical task where, say, if models had well-calibrated confidence scores, you could tell the model something like, "If you're at least 60% confident, give me the answer." As long as you had that core requirement of reasonable calibration, that might be fine. Basically, if you ran that thing 100 times, the model should get the answer right 60% of the time if it's 60% confident.

That might be fine in creative-writing settings, or it might be what you want in particular settings. Whereas if you had a really complex, really critical task, you might be able to say something like, "I can't afford for you to get this wrong. Think as much as you absolutely need to. If you're in any doubt that the answer is wrong, tell me. But if you're going to output an answer, make sure it's right."

Obviously, we're far from any model that can do that. But that kind of behavior—thinking about the problem from almost a user-interaction perspective—I think is going to be extremely interesting over the coming years.

I agree. The big challenge is just building the apps and imagining what can be done with this technology. Just the basic stuff, like doing enterprise search really well—we've got a long way to go, but it's going to be an exciting journey.

For sure.

Max, thank you so much for joining us today. It's been amazing.

Thank you. Likewise. Thanks for having me.