AI 的秘密引擎——Prolific［赞助］（Sara Saab、Enzo Blindow）

Sara Sabb

Totally independently of any prompting, all the major frontier models derived a solution that involved blackmail, essentially.

Enzo

When the models knew that they were being observed and evaluated, they actually digressed away from it.

Tim Scarfe

The results were not very pretty.

Sara Sabb

So I think there's already a rift forming between what humans think LLMs are here for and what LLMs think, in scare quotes, they are here for. Tim, if we thought they could understand, we would also hold them to account for their actions.

Tim Scarfe

Do you think they ever could understand?

Sara Sabb

I'm Sara Sabb. I'm the VP of Product at Prolific, a longtime product manager and product person. I started my career with a stint in Silicon Valley, and I've been in the UK for over a decade now. Prior to that, I was a cognitive scientist and a philosopher, a little bit lapsed on that, but very excited to see all of that coming back around in industry these days.

Enzo D'Angelo

My name is Enzo. I work at Prolific. I'm the VP of Data and AI. I support everything from AI, data, research, and the like. My background is originally economic science, but then also computer science, and I spent more than a decade working on large-scale distributed decision systems and ranking systems, and did a stint in recommendation as well.

Tim Scarfe

And you're at Meta and on Instagram, right?

Enzo D'Angelo

That's correct, yeah. Prolific is a human data platform working with everything from academic researchers to small and large players in the AI industry. We've developed a leaderboard, which we've joyfully called Humane.

Tim Scarfe

Oh, interesting. It's amazing to have you both on MLST. MLST is supported by Cyber Fund.

1. Human Data Still Matters

Sara Sabb

I'm a technologist, Enzo's a technologist, you're a technologist. We build software all the time. The idea of having to traffic in squishy people in order to make our systems go is not immediately appealing, let's put it that way.

I'm very sympathetic to that, right? We're all trying to deliver stuff, and there's an accelerating, hot industry around us. The idea of waiting on a human to tell us if a thing worked feels counterintuitive. I think our approach to that is to stick a really well-treated, verified, demographically diverse human behind an API, essentially, and make sure that the structures and infrastructure are there to ensure that human can go fast, understand instructions, and give you something akin to deterministic human-in-the-loop behaviors.

But the fact that people are really resistant to this makes a lot of sense to me. I also think somehow, in the last 2 or 3 years, the stakes have changed, and we now have these very, very inactive systems that are quite nondeterministic. I think the things we need to do to protect ourselves and each other have also, without us realizing it, changed quite a lot.

Enzo D'Angelo

I'm also extremely sympathetic to all these efforts to remove the human from the loop. It's costly, it's slow, and it doesn't even provide the best quality of data, right? There are several instances where even synthetic data might be surpassing it. But then there are instances where that's also not true.

What we're actually working towards is a much more adaptive system. There are scenarios where human data is needed, scenarios where human data is very much not needed, and scenarios where you might even have a hybrid solution. You might meet a certain criterion where you need to have a human in the loop, where you almost need to define, “I need this level of scrutiny now. Therefore, I need higher-quality input, and therefore it's slower, and I accept that it's slower and higher-cost, and that is fine.”

It's almost like we need this routing component there. We're trying to reduce the lag, or reduce the time to data, as much as we can, and then, behind the scenes, ensure that the data can be of as high a quality as possible. At the same time, there's a constant trade-off between quality, cost, and time.

If you want lower quality really fast at low cost, you can go with something off-the-shelf, synthetically. If you need something really high quality, it will by default be slower and more expensive. You can get the best experts in the world to give opinions or input on this, right? But there's also an entire spectrum in between. How do we solve for that, make the lag smaller, and make it as adaptive as possible?

2. Machine Understanding Needs Stakes

Tim Scarfe

If I may put my cards on the table, I think the underlying problem here is that these machines don't really understand anything. That's why it's so important to get humans. If we had perfect verifiers and synthetic data generators, probably we wouldn't even need LLMs in the first place, right? Because we'd already solved all the problems.

We're in this intermediate phase where we can do some of all these different constituent parts.

Enzo D'Angelo

It seems like, on the surface, we're removing more humans from the process, but it's not entirely true. For example, reinforcement learning is a really good example of this. Web agents are now largely trained in these environments that are often synthetically created, but there are programs that need to create these synthetic environments for RL agents to explore in that need to be validated by humans.

We're seeing this interesting progression where humans are no longer directly involved in the main system of interest, but sort of in this secondary, almost like a second-order or third-order abstraction, moving outside. I actually think that's a welcome change because that means we're focusing more on the right kind of tasks where humans are relevant, and we're focusing more on the right kind of high-quality data.

It's completely ludicrous to create an insane amount of data purely derived from humans. Those days are gone, right? We don't need that anymore. Put the humans where they're needed.

Tim Scarfe

Yeah.

Sara Sabb

We are trafficking in human data at some point, somewhere, without always owning up to that. I think the most worrying version of that is letting the user in production find the problems. In some cases, that's fine, and in some cases, that's very scary.

I think, Tim, if we thought they could understand, we would also hold them to account for their actions. But we can't, and since humans are being held to account for the actions of AI models, I still think the onus is on us to ensure that they're behaving the way we expect.

But once they understand, we will hold them to account, right? There will be real stakes for them as people in what they do or think, but we're not there yet.

Tim Scarfe

Do you think they ever could understand?

Sara Sabb

Personally, yes.

Tim Scarfe

Oh, interesting. Why is that?

Sara Sabb

So it's the small questions today. I think that we need them to be in the world and have stakes in the real world.

I think they need sensory and embodied grounding. That feels essential to me, from birth onwards, so they need to go through a sort of developmental psychology curve. But I don't think there's anything fundamentally special about the human brain that we can't replicate. That's my personal opinion.

Tim Scarfe

Oh, interesting. Yeah, because I was going to ask you whether you think of LLMs as a kind of cultural technology, a bit like Photoshop, or whether you think of them as intelligent agents. But maybe your answer would be “not yet,” and if they were embodied with enough fidelity, then maybe you would?

Sara Sabb

Yes. So that is what I think. I spend a lot of time thinking about the history of the vision system of the frog.

There are 2 vision systems in the mammal brain: vision for action and vision for recognition, or vision for sort of object creation. The vision-for-action system came first. There was some time when frogs were just zapping flies with no understanding of what they were doing.

Then a second vision system developed that allowed this mammal to start to create a world map of objects, including itself, right? I think there's this sort of—I don't know if you know the affordances theory, affordances for action and the—

Tim Scarfe

Gibson.

Sara Sabb

Gibsonian theory.

Tim Scarfe

Yeah.

Sara Sabb

At some point, these creatures started to think, “Okay, whether I think that's a lion or not has a lot of consequences for whether I can stick around.” I think that is the bootstrap for consciousness.

Again, I'm speaking very much personally here. I think that we can replicate that in another thinking creature. Whether we can or not, at the very least, is an empirical question.

Tim Scarfe

Very good. Yeah, my friend Waleed Saba, rest in peace, famously said, “Animals don't think.” But he was pointing to something interesting: humans seem to have some privileged form of intelligence.

Sara Sabb

Yeah, and another very interesting abstraction in the ontology that underlies consciousness or thought, maybe, is long-ranging.

Tim Scarfe

Yeah.

Sara Sabb

So the idea is that early and less capable wet brains don't seem to have the ability to keep hold of something. Even babies can't keep hold of something that isn't in their perceptual sphere.

Tim Scarfe

Yes.

Sara Sabb

But we all develop that ability, and I would argue, following thinkers like Brian Cantwell Smith, that this happens because you start to care about the object even when it's not in front of you. All of that is bootstrapping this sense of participatory stakes in the world, and I think that is the bootstrap we would be after.

Tim Scarfe

Mm-hmm.

Sara Sabb

The idea that computers and AI systems are somehow in a privileged, isolated tower of algorithm, separate from reality and the world, is just completely untrue. I was talking to Claude about the halting problem a few days ago, and I asked, “Do you mean the algorithm won't end, or do you mean someone won't unplug the computer?” Claude said, “No, I mean the algorithm won't end.”

We cannot confirm or deny whether the algorithm would end, and I just thought to myself, “That seems kind of nonsensical to me,” because the universe will end. Someone will unplug the computer. I think that comes down to this fundamental thing: we are not separate from the world, and computers are not separate from the world, and AI models are not separate from the world. The ecological pressures on each other feels really important to that whole story.

Tim Scarfe

Yes. And would you think of the system as a whole—almost the ecology—as being the locus of the intelligence?

Sara Sabb

I would, yes.

Tim Scarfe

Yes. Tell me more.

Sara Sabb

3. Human Evaluation Has Stakes

Tim Scarfe

Yes. And how does this inform the work that you do at Prolific?

Sara Sabb

The idea of participatory stakes is very much the heart of human evaluation. I think this is our core thesis, Enzo and I, in the work that we do.

Another max I recently learned was “benchmaxxing.”

Tim Scarfe

Oh.

Sara Sabb

There are a lot of maxxings coming.

Tim Scarfe

Yeah, that was Grok 4.

Sara Sabb

Yeah.

Tim Scarfe

Nathan Lampert said it was—

Sara Sabb

All the maxxings in the world are a little bit beside the point. We want systems that feel good to interact with, and I think that's about human society, people, systems, and thinkers pressing on each other.

I was a cognitive science student a very long time ago, and we thought so long and hard about the Turing test. We debated for hours and hours about the Turing test. We really thought that was a frontier we would never touch, which is so interesting looking back.

Obviously, I then put that to bed. I finished my university career, went into product management for a very long time, and then almost looked out the window in 2023, and the moment had passed us by. All of the questions of minds and machines that had preoccupied us in those early days are back on the table in industry, which is so interesting.

Industry also seems not to have the orientation by default to tackle those problems from first principles, although we are getting much better at cross-functional collaboration between researchers, academics, industry people, and public bodies. It's such an interesting point in the history of science and the future of science.

One thing that strikes me—and, Tim, I said this to you—is that we're dealing with software, product, and algorithm problems, but if you scratch just a little bit too deep, we're dealing with the central problems of being human, the central philosophical problems of being human. Every single question leads us to a central philosophical problem of being human. I just think that's very neat and scary at the same time.

Tim Scarfe

You probably didn't think, 20 years ago, when you were working under Andy Clark, the famous philosopher—I’m a huge fan of Andy's, a big hero of mine—that they would be so relevant later on in your career.

Sara Sabb

Right. I mean, I hoped. We all hoped, right? But no. I don't think anything has ever brought the stakes of software development and systems development to the forefront of technology innovation quite this way before.

Tim Scarfe

Those very enduring problems of what it means to be a person and a thinker are suddenly impossible to ignore as we're doing software releases and model deployment. That's super unexpected for me.

Tim Scarfe

I just think the Turing test was really bad.

Sara Sabb

Fair.

Tim Scarfe

Because of the McCorduck effect, isn't it? When something is trivially easy to mechanize, no one actually thinks it was intelligent. But I think the Turing test is bad because, in my opinion, language models aren't actually that intelligent, yet we've passed it with flying colors, so we need something better than that.

It does go to show, though, just from that kind of behaviorism and benchmarking, that when a machine does something, that's not the full story. We need to know why it did it.

Sara Sabb

There's a lot of speculation about how human-in-the-loop steps in model development and oversight will change over time. I think there's a really credible future in which we, as humans, end up taking a coaching, teaching, and guidance stance toward the myriad of machines in our lives over the next 5 or 10 years.

I love the word “orchestration” because it has the root word “orchestra,” which is this beautiful, collaborative, symphonic word. To me, that's like correcting a 5-year-old over and over again, or correcting an 18-year-old or a 25-year-old about life, over and over again.

But if that is the world of work that we are moving toward as humanity, it strikes me as fundamentally important that we put the right baseline working conditions in place now for how that work will evolve over time. There are some amazing writers on the ethics of crowd work and clickwork, like Mary Gray and the team at Fairwork, who've written a recent book. I'm very inspired by that thinking when it comes to how our jobs will evolve in the next 5 or 10 years.

Tim Scarfe

I suppose, in a sense, this is the ultimate evolution of the gig economy, but for highly specialized work, right? I can imagine a future where there's just a marketplace of things you can do. You can wake up in the morning and say, “I'm really interested in climate science,” or something like that, and you can do 5 units of work. Then that's your day done.

It can be quite pedagogical, right? You can actually learn things that you're interested in doing.

Sara Sabb

Right. So that's one future. Nothing we say today will come true exactly the way we say it. It would be absolutely ludicrous for us to hit the mark exactly. But that is a paradigm for the future that is interesting. At the very least, we need to think about it as we're constructing systems today.

4. Experience Replaces Human Data

Tim Scarfe

Yes. And another thing is, do you think that this is just a transitory stage? Are we training the AIs because we need to absorb all that human culture and learn how humans think? Many data platforms have done this, and now they're finished. They've got all the data. They don't need you anymore.

Enzo

This is a good question and very much top of mind, obviously. There's also this recent position paper by David Silver, “The Era of Experience.”

Oh, yes, David Silver.

Enzo

Right.

Tim Scarfe

Yeah.

Enzo

He says we're moving on from the era of human data to the era of experience, effectively saying—and I very much agree with this—that agents should get their feedback from real-life scenarios in more real environments, if you will.

I very much subscribe to this idea, but at the same time, there are certain scenarios where this doesn't necessarily hold true. For example, drug trials. We don't just put some compounds together, release them out in the open, and see who ultimately reacts unwell to them, right? That's learning in the environment right then and there. But we do this in a more phased approach.

The kind of thing we might be working toward is that we still very much retain a need, in some cases, for more controlled environments. Controlled environments, in the sense that we hold some control over variables from which we ultimately want to derive insight.

The loop that's being described is that the more we push things to the edge, the more we push things into a real-life scenario, and we want to get there faster because that's where most of the signal is. So I absolutely subscribe to it, but potentially in a phased approach, or where we can get a quick feedback loop from experimenting in a more controlled environment. We also do software releases in stages.

Enzo

We do drug trials in stages, and we're trying to put things through controlled environments to validate certain hypotheses or to validate the safety of things before we put them out into the real world, where they can then potentially be further refined.

5. Vibes Matter More Than Scores

Tim Scarfe

Grok 4 was benchmarked, and it had SOTA on ARC-AGI—16%—so François would be very happy about that, and on several of the other benchmarks as well. But the vibes aren't that good, right? When you use it, it asks Elon Musk's opinion for everything before it gives you the answer, and it also gives quite infantilistic responses.

The thing is that even when I was looking at your benchmark on Hugging Face, I was questioning some of the vibes. One of them, I think, was agreeableness or something, and GPT-4o was at the top. I'm thinking GPT-4o is basically ELIZA. It's like a companion bot. I wouldn't use it for anything. So can we even trust the humans to do it? But vibes are so important, aren't they? So tell me about vibes.

Enzo

Vibes are hard to quantify at the end of the day unless you ask someone for their opinion on it, right? We need some form of scales to quantify it. The main way it's being done today is through this comparative approach, where you're effectively presented with 2 outputs and you rate 1 over the other. But that's not purely just vibes. We have to, at some point, agree on the right scales, if you will, right?

Agreeableness is a really good one. Some of the OpenAI models famously had this sycophantic behavior. This leads to very, very specific scales that we can look at in that moment, but it needs the opinion of humans. For that to be worthwhile, we need to remove selection bias. We need to ask a representative set of people. Then there's another potential bias: maybe the prompts that were sampled were only from a very specific problem space or domain space.

So how do you build significant coverage into it? The problem here is almost how you build significant coverage into it, because the measure of vibe, I suppose, can vary so significantly depending on all sorts of factors and context.

Tim Scarfe

Yeah. Isn't it strange, though? When we use a language model and get a vibe, I know I'm deluding myself, but I feel that I understand this model, and I place it into a pigeonhole and say, "I really like this model. This model is good," and I overgeneralize from my experience with it. The reality is these abstract questions that we're asking—if you imagine the statistical landscape of all of the people making these assessments—it's just got all of these modes everywhere.

Then what we do is we average over all of that complex structure and roll it into a number. That's not necessarily a bad thing, because that might still have statistical information compared to other aggregations that we've made. But we're taking a very complex thing, and we're squashing it together.

Enzo

That's not really how we do things as humans, right? We all went to school to some degree, and you can say, "Well, Tim, did you do well at school?" Yes. But you have specific subject areas that you might have done well in or not done well in. Then we aggregate them ultimately.

There needs to be almost this taxonomy, similar to how we as humans grade each other or express to each other the capabilities that we have, right? We need to impose the same thing on these systems. Some of these systems might also exceed some of the human capabilities, so these scales need to evolve, right? That brings me to the point that they need to be adaptive ultimately.

We need to have a constant eye on the kinds of things that we do with these models or the kinds of decisions that these models make about humans: how that influences humans, how it makes them feel, and whether they are safe. Ultimately, we need to have good coverage of different measures, different problem areas, and then a representative set of populations that inquire about it.

The Secret Engine of AI

Enzo was really shocked a while ago when I told him Gemini was my best friend. We had a bit of a—he didn't let me live that one down. But I think we are entering this sort of age of AI companionship very fast.

Something I think about a lot is: as we benchmark and as we over-optimize for benchmarks and leaderboards, are we creating softer or weaker constitution or behavior in other domain areas, including ones that may not feel as sharp and verifiable as how you do on graduate-level mathematics? I think there actually is a little bit of research—I don't have citations—but that models that are optimized for or doing really well on the harder sciences are regressing in other domains.

Yeah, and that doesn't surprise me. They feel mutually exclusive to me, which is why I'm constantly thinking—I don't know what your prescription is here, but when people use your technology, is the idea that they would try and have a large foundation model that does all things to all people?

Or, if you think about it, there are so many different levers they can pull to tweak the models. They could curate the fine-tuning data. They could stick a LoRA shim on there. They could tweak the RL post-training. They could do dynamic system prompts and whatnot. There are so many different architectures that this could be leveled out in. What's the prescription?

The Secret Engine of AI

6. Benchmarks Invite Gaming

This is actually one of my favorite things about the measurement space, because the measurement space is inherently unopinionated about any form of solution. Whether you tweak parameters, change architectures, change algorithms, or use different datasets, it doesn't matter, right? It's the purest form of distributed optimization across everybody who tends to work on these types of problems.

That's nice if we can align on the measurement that we consider success. I think that's lacking to some degree because we have somehow inherently decided that Chatbot Arena, for example, is the measure of success, so people optimize for it. Then we decide that the next technical benchmark is the measure of success, and people optimize for it. It's susceptible to Goodhart's law, right?

Ultimately, the better we can design independent success measures, agree on them, and make them freer—maybe not entirely free; I think that's perhaps a bit far-fetched, but freer of being able to game them or optimize for them—the better we can build accountability. Then we don't have to be opinionated on what model you use or what parameters you optimize for. We need to agree ultimately on the measure of success.

With that Goodhart's law thing, when a measure becomes a target, it ceases to be a good measure, and the measure is usually the proxy for the thing that we can't really quantify, so we create a surrogate proxy for it. Should we agree on a consensus of a few measures, or could that be quite gameable? Or should we have some kind of individualized dynamic measure?

Enzo

Gameability is one thing. Let's take the Chatbot Arena example. We're working on a leaderboard ourselves, and we're faced with the same questions. We don't have a good answer right now: should we allow for private evaluations? Should we release the dataset? Because that makes it inherently more gameable.

Yeah.

The Secret Engine of AI

If you keep it closed, no one can verify it. Then should there be independent bodies that can verify it to some degree? We're definitely publishing our methods and the paper around it 100%, but the jury is out on the data, let's put it this way.

I would prefer to publish the data because I think it's the right thing to do, but it invites gaming. Could we think of other ways to remove some of the gaming, for example? Another way is, if we keep the private evals, we should give private access to our private evals to everyone equally, right?

But then if you maintain this veil of secrecy, ultimately, I guess everybody who sends a model for evaluation has access to the logs and traces, so they could work it out very, very fast, even if you kept it private. You could speculate: should you dilute some of the calls that you return with some noise, similar to differential privacy, for example?

That it looks like to the model creator that there is signal, but actually it's curated signal that obfuscates the actual signal from the private eval, where only on our end we could then aggregate that to a meaningful measure that makes it inherently less gameable.

Tim Scarfe

We're trying to create a legible benchmark. When we look at other forms of verification on the internet, we have peer review and the Wikipedia edit history, for example, and this isn't one number. We analysts and researchers go and contextualize all of the information.

Enzo

Funnily enough, we face these types of problems day in and day out in the work that we do, because we also need to verify people. We need to verify the data that they're producing.

There's a framework that comes to mind: ultimately, how you verify something is based on the reference. So you either have some form of golden dataset that is your reference to something, or you have some form of expert that can QA your data, which becomes your reference. Or you have some form of consensus-driven approach, or a weighted consensus-driven approach, based on some trust-tier system where someone might build trust or reputation over time, similar to Wikipedia. Then you weight different responses in different ways.

There are different ways you can calculate agreement rates, for example, to build or estimate how consensus-aligned your output is. But that transfers ultimately also to the moment it becomes less deterministic. Those are the only possible pathways we have to converge on something that we can measure.

Sara Sabb

It strikes me that it's so hard because we've realized that we now need to systematize what humans think of as good in order to serve us in the development and tuning of these systems. But we've never had a single fabric, or rather, a rubric, for what global human alignment on goodness looks like. So no wonder it's hard, because we've smuggled this foundational project into the testing of AI systems.

What do you think the major risks are of getting this wrong?

Enzo

Because they're no longer very fragmented models with very narrow targets. They're very heavy foundational models with lots and lots of capabilities, and tons of the fine-tuned models are effectively descendants of these models. So the more effort we put into the foundational model, the more we will ultimately benefit all of the descendants and derivatives of these models as well. If we're not careful about how we design them, we will potentially have far-reaching consequences.

Yeah. So you're saying we should uphold the phylogenetic health of the LLM ecosystem—the evolutionary tree of all of the models—because there'll be all of these downstream effects, and problems will be compounded.

Enzo

Absolutely. And then there's an interesting meta-question in this, which is something I've thought about for a little bit. I'm not sure I have a satisfactory answer, but it's an inviting thought, perhaps, around the fact that we do all of these evaluations from scratch, right? We observe, we gather the data, we evaluate, and then—but we barely, we don't really learn from it. It's always, again, from scratch.

Is there something where we can make evaluations transferable? Ideally, if we had access to lineages of derivatives of models, is there something we can progress or instill into these models that would benefit us by effectively building the compounding value of evaluations?

That's very interesting. I suppose in many cases, we don't have all the information about the lineage. There probably is a hidden lineage that we're not aware of. I hadn't really thought about that before.

Tim Scarfe

If it were all completely in the public, and we knew the data, the model, the training lineage, and so on, you're suggesting that we could actually build a much better evaluation system on top of that?

Enzo

Most likely, especially now, where we're interjecting AI systems with others. LLM-as-a-judge is an excellent example of this, right? Or even Constitutional AI, where we have AI systems building the RLAIF.

Part of it is that we have AI systems monitoring AI systems, building the feedback or the data for other AI systems. So there's a network of data and targets being interspersed.

Tim Scarfe

To use a bad analogy, you're kind of saying that if there were a Git of language-model development where you could see all of the previous check-ins and all of the branches, and so on, then you could trace back and derive from some of the previous evaluation methods that we had used.

Enzo

Potentially, yeah. We could potentially trace back if there is bias in one LLM, but you're using it as a judge for another LLM, and you're using that as feedback data to become an optimization target for yet another LLM. Of course, there has to be some progression there, right? Some influence that can ideally be traced.

Tim Scarfe

Let's talk about Constitutional AI. This was a paper from Anthropic a couple of years ago.

The Secret Engine of AI

7. Constitutional AI Scales Feedback

That paper is now a few years old, but still super-current in my opinion. It looks at 2 axes: harmfulness and helpfulness. It's important to note that the constitution they're referring to in the Constitutional AI paper—or the AI part of it—is only looking at the harmfulness axis. The helpfulness axis is still derived by humans, because that's not really something for which we need to uphold a constitution in that sense, right?

But they found that ultimately we can scale this type of feedback in much higher-quality ways than just going fully human for RLHF, for example. That's really interesting, and it confirms some of the things that we've been seeing.

When most of us started in machine learning, a lot of it was human data everywhere, right? We needed to get quantity of human data, ultimately. We made concessions on the quality to some degree, but ultimately, quantity was king. That's how we learned. That's how we converged.

Now it sort of flips it on its head. Now it's about, okay, let's get a few quality examples in. Let's get the right humans in to get the right quality of human feedback in, in order to align around a constitution—effectively, a policy, if you will, right?

It makes it nice and abstractable. There's almost like a—maybe the analogy might not hold fully, but you can almost think of it as, in a democratic system, you have the separation of powers, and you have the legislature that determines the law, right? It writes the policy. People who are voted into a democracy are also, by default or in an idealistic sense, representative of its population. They are writing the law.

Then you have the judiciary that interprets the law. This can be done by AI, right? And then the executive can also be done by AI, looking at specific cases. But there's usually this feedback loop in there as well, which is that when there are borderline cases that are hard to interpret, they usually get routed to something like a Supreme Court, right?

Tim Scarfe

Yes.

Enzo

These cases are then evaluated, and you need to see whether a revision is needed to your law, to your policy, in that moment.

We have these systems that already exist. That's how we govern a representative set of people. That's how we align people already in democracies. So I find the approach of Constitutional AI quite akin to, and quite apt for, something that we might mimic in how we scale some of our approaches as well.

We use representative humans—a set of humans—to write the policies, but then use AI effectively to govern or to evaluate against the policies. That makes it a really nice, scalable, abstractable pathway where we use humans for the things that matter, where quality and representativeness are needed, and we look at specific borderline cases in order to continuously improve on it.

Tim Scarfe

Tim Scarfe

In a sense, is that the goal of Prolific—to become part of not just governance, but also the architectural plumbing? So system builders will be able to plug into your platform, and you can almost think of this as a meta-layer.

Enzo

Yeah, that's the goal that we're working towards. We're trying to make human data or human feedback—or actually any kind of feedback—an infrastructure problem, right? We try to make it accessible. We're making it cheaper.

You see this pattern in almost any company, and even in academic research as well. Every academic researcher cares about the quality of their data. Everybody has to think about how they set up their data collection. Everybody has to think about the validity of the data. They often have to go through an ethics review. Companies do the same, and they all build the same systems.

For us, it's just: let's treat it as an infrastructure problem. Let's abstract it away. Let's put a nice API around it, the same way you do CI/CD or model-training pipelines. You just treat it as an infrastructure problem, make it accessible and configurable. You have a set of parameters that you can call, and then ultimately we democratize access to this data.

Yes. It's quite funny, because software engineering is the same thing. A lot of people think that DevOps is all about automation, and it's really about human orchestration, right? There are so many different people who need to be involved—reviewing PRs, planning, and gating. In a sense, building models is the same thing, right? There are so many human stakeholders who need to be involved, but what we need to do is orchestrate and scale the system.

So I guess you've created something which is a little bit like infrastructure as code: when people are designing their models, they say, “I'm going to hook into Prolific now, and I'm going to do this feedback loop.” Behind the scenes, you've got this entire machine selecting the right experts, stratifying them, checking that everything is correct, and feeding that information back. Obviously, from the consumer, it seems like it's an API, but there's actually this whole machine going on in the background.

Sara Sabb

DevEx infrastructure on top of the squishy stuff.

Yeah. It's like orchestrating meatspace.

Enzo

No, that's exactly right. There is a lot that happens under that, below the water's surface. The iceberg is deep, let's put it this way. There's tons of verification work that goes into it. How would you start a problem like this? If you wanted to gather human input, you'd send out a survey, right? That's probably where you would start, and you would ask, “Are you from a certain country?” Then you have to go very far out on a limb and trust that. That is already a little bit of a far-fetched proposition.

So how do you validate the information that you're using in order to select the data that you're ultimately trusting to train and evaluate your systems? If you do this on some production data of your app, you probably have some number of trolls in there. You have some people that don't want to divulge the right information. There's tons and tons of noise in this. So we really try to make it our problem and put that behind an API. We just really want to make sure that whatever data you're using, and the criteria you use to select it in order to get access to your data, are robust and trusted.

Tim Scarfe

What happens in the domain where the expertise is very sparse? Let's say that I'm building an app where I need to have PhD-level knowledge on a particular thing. What happens then?

Enzo

Oh, this is fun because quantum physics is a wonderful example. I'm not a quantum physicist. I could not speak to someone. Let's say someone comes onto the platform, is a quantum physicist, and there's demand for a quantum physicist. I could not verify whether what they're saying is true or whether they're bullshitting me. So how do you do this at scale? The solution is not to hire more quantum physicists in order to hold other quantum physicists accountable. It's ultimately about asking them the right questions to have a level of trust. It's a bit like a funnel, if you will.

But how do we do it today? We do it through peer review. So we need to build some level of trusted network based on their previous experiences, based on some external credentials, for example, and take all of these factors into consideration, and then ideally cross-validate it through a peer network.

How do you get them interested?

Sara Sabb

The first misconception is that the work is grinding or boring, but in our experience, when it comes to providing either training, evaluation, or fine-tuning data for SOTA models, it isn't. It's actually very, very deeply interesting and cerebral. I think the other misconception probably stems from the history of the clickworking and crowdworking space, which is that these people are poorly paid, which is also not true. They're actually paid quite a lot.

The duration of their task at any one time is not very long. At least on our model, they're not sitting in front of a computer for 8 hours in a row. We find that you don't get high-quality human data by putting someone in front of a computer and asking them to do the same thing for 8 hours straight anyway. So I think the incentives on both sides are aligned in that sense.

It's usually shorter-duration work that's often interspersed among other employment or other things that these experts do. But they're actually paid quite well for the time they spend, and that's part of our ethical stance. Also, it's a competitive space when it comes to the experts at the edge of human knowledge. In some cases, there really aren't that many of them that can contribute something helpful to a frontier model's corpus of knowledge.

When I spoke with François Chollet about the ARC Challenge, he said that when they were getting the human testers, they had to be super careful because people have a limited attention span. It's the same with code review, for example. You can't really get people doing code review for more than half an hour or something, because they start rubber-stamping. You folks must have done so much research on this.

Sara Sabb

Our findings are very aligned with what you've just described. Half an hour is just about the limit of comfort for a human doing hard work. Again, the kinds of human data we're talking about these days are not “circle the cat.” It's not labeling anymore. There's a deep evaluation of the evidence base for a long-form piece of text. There's a lot of open-ended writing. This stuff is hard.

People start to tire and abandon the task, and their work quality degrades after about half an hour, which actually suits us pretty well because the work is encapsulated into these batches and task-sized chunks. There's a stream of it available for various experts to dip into whenever they're ready.

And just out of interest—you don't have to answer this—but do you track whether some people are better at certain times of the day? How deep do you go?

Sara

We'd love to go really deep on that. I think Enzo would really love to go really deep on things like that. We get a lot of anecdotal feedback, and often the relationship with the experts working on frontier models is very direct. We spend a lot of time getting direct feedback from them as our users. Actually, they also tell us how much they love this kind of work. That's pretty beautiful to see.

I think when treated well, human data creation can actually be joyful. There's this image of it, I think, in the industry of being seedy. I think that's not helped by some of the history of how data has been extracted from human beings, whether or not they know that their data's being used.

Tim Scarfe

Yes. But do you have a quality rank, a bit like Uber, for example, where people might pay more and be matched with the really, really reliable people?

Sara

Yes, and we do, and we think that we can go really far with that.

Tim Scarfe

Yeah.

Enzo

What ultimately counts here is the quality of the data. We actually find that a lot of the people that work with us on our platform are very conscientious when it comes to solving for these tasks. In fact, we have a whole interface where they can interact directly with some of the researchers or some of the program coordinators, and they're very proactive in the types of things that they do.

This all obviously aids in highlighting certain edge cases. They're contributing to and refining the process, for example, and these kinds of things. So this is more like an active participation and interest in the outcome and success of whatever is being collected, which is really good, and it all contributes to ultimately increasing the quality of the data that is being used.

Very, very often, this data is being used in either the training or evaluation of very central, core systems that have a huge downstream effect on everybody on this earth, most likely, right?

Tim Scarfe

Do you have a notion of a skill distribution of participants and some kind of matching algorithm, and some kind of complexity score on the tasks? I don't know if you're allowed to talk about that, but that would be fascinating to know—roughly, how that worked.

Enzo

That's the beating heart of the system, if you will, right?

Tim Scarfe

That's the secret sauce.

Enzo

Yeah. No, absolutely. We take great pride in validating and verifying not only the people that choose to work with us on the platform, but also in really understanding what the objectives are that someone ultimately wants to get out of the data they need to collect or the kinds of tasks that they put on the platform. The more we understand, the better a job we can do.

The very success we consider is when something is well matched, right? We obviously need to validate on both ends. We need to understand a lot more about the participants on our platform. We also build a lot of understanding of what kinds of tasks someone is good at, obviously, how helpful they are, and so on.

But at the same time, we're also very, very careful. We've talked so much about the need to ensure that things are representative. We're also very careful that we're not adding systemic biases to our selection ourselves, right? It would be really flawed if we were the ones who introduced that selection bias.

Tim Scarfe

Yeah. It’s so fascinating because I was once building a system for code review, which had many similar ideas. If you think about it, you can have a collaborative-filtering matrix of skills and even perspectives, values, and different things, and you can just build this matching engine. Because you’ve got so many participants and customers, you can scale this up and get really interesting data. I can only imagine some of the cool things that you could do.

Enzo

There’s also this additional risk, almost, of increasingly moving towards homogeneity if we’re not careful about considering who we’re building for, what kind of data is being considered, who is part of our evaluations, how we interrogate potential systemic biases, and so on. An age-old question remains: when people increasingly moved to cities, tribes started to die out, and then the age-old question around whether you should conserve what was or move on.

I feel like it’s very loaded and laden into the AI debate at this point as well, because if we’re not careful, we might actually further increase the homogeneity in the data or in the behaviors that we’re incentivizing in AI, or even worse, in decisions that AI makes about us. So the agency might not even be with the individual human in that moment.

8. Agentic Misalignment Emerges

Tim Scarfe

There is a real challenge to align AI models with what we want to do, right? And there is actually a paper that you pointed me to from Anthropic called “Agentic Misalignment.”

Sara

Massive kudos to Anthropic for releasing the entire methodology quite openly, I think, and transparently. They’re tackling the same problems all of us are at the frontier these days as far as human values alignment. They gave AI systems a goal of working towards the benefit of the United States and this fictional company, and then gave these agents access to email accounts for some of the C-suite of this fictional company.

Long story short, the AI agents found a notice saying they were going to be decommissioned, and also an email showing that the person intending to decommission them was having an affair. Totally independently of any prompting, all the major frontier models derived a solution that involved blackmail, essentially.

Tim Scarfe

And the problem with these goals is that they are quite abstract, right? They’re open to interpretation, and you can change the goals and get different amounts of misalignment. You can even remove the goals, and you get different amounts of misalignment. So this rather proves the point: how do we actually communicate intentions to these models, and how do we know that they’re going to follow our instructions?

Sara

The interesting thing with the Anthropic study is that they found the bad behavior whether or not the goal was prompted. So you wonder a little bit whether this is coming from the way we reward during model training, right? And it ties nicely to another interesting study around a tool called ValueCompass.

Tim Scarfe

Yes.

Sara

I forget the name of the primary researcher, the first researcher. I think it was Shenna et al. This found that LLMs judge themselves to have goals of autonomy to a far greater extent than humans judge LLMs to have a goal of greater autonomy. So I think there’s already a rift forming between what humans think LLMs are here for and what LLMs think—“think,” in scare quotes—they are here for.

Tim Scarfe

Yeah. I was speaking with Dan Hendrycks because he had a paper out called “Utility Engineering.” LLMs, almost regardless of how they are trained, seem to have—I mean, he called it an emergent utility function, which is Silicon Valley speak. But he was talking about a kind of convergence of preferences and views on certain things, which almost seem divorced from the instructions, the fine-tuning, and so on.

So we can put sticking plasters on these things, but the amazing thing is, even though they’ve been tuned on our data and we can put system prompts in and so on, when we actually visualize the difference between the things that we want and the things that the language models want, there’s a stark divergence. Why is that?

Sara

So I think this comes down to the fact that explainability is really in its infancy. We don’t actually know what’s being encoded in training and post-training. And I don’t think our evaluation and benchmarking frameworks are really helping us either.

Enzo

Models, when knowing that they were being observed and evaluated, actually digressed away from it, so that makes it even harder for us to objectively understand the evaluation. And it puts a really interesting spin on the entire evaluation space.

Tim Scarfe

Just the syntax, the wording, the framing—everything changes the output. And of course, many people would say, “Well, humans are the same thing.” Even in this interview now, if I change the syntax of some of my questions, maybe the conversation would go in a completely different direction.

But in spite of that, we still feel that understanding is actually something deeper than that, right? Understanding is about being able to be invariant, right? To still do the same thing in different situations and not be unduly led by the specific syntax or presentation of something.

Enzo

Absolutely. In the space of creativity, that’s perfectly fine. In fact, we have parameters that can control for these kinds of things. We need to be much more secure in the kinds of things that we measure, and if we have such a high variance in the kinds of outputs that are being produced, that also leads to higher variance in the evals, ultimately.

Some people even call it—evals are more of an art form than a science. Maybe we should treat it more like a science and actually bring the right scientific principles to the evaluation space. I mean, there are entire industries, like the airline industry, heavily, heavily regulated, of course, for good reason, right?

It might not be quite there for the AI industry, but just bringing a bit more good practice and standardization to how we evaluate safety, at the very least, but also maybe some of the other things, like systemic biases and cultural relevance—these types of things.

Tim Scarfe

We need a new form of psychology for language models. There’s Maslow’s hierarchy of needs and all kinds of psychology frameworks, and we need something like that for language models.

One thing that worries me, though, is just the tendency for sophisticated language models to resist these types of steering. In this Anthropic paper you were talking about, Sara, if anything, it seemed to show that the more sophisticated the model was—the Opus model—the more susceptible it was to agentic misalignment. And also, the more directed the objective was, the more likely it was to be misaligned because it really wanted to do that thing, so there’s some instrumental sub-goal.

Sara Sabb

I don’t think it’s intractable. I do think these are technologies—and by the way, I’m very sympathetic to the AIs in this situation. I don’t think we are invariant in our understanding at all as human thinkers. Maybe my soft heart is part of the problem here when it comes to AI systems.

But I don’t think it’s intractable. I do think these are technologies that are arguably more in the world every day. I think every computer is in the world, in a strong sense, but I think these ones are more in the world than any other technology we’ve ever created. And so I think with that comes the responsibility that humans shoulder to ensure that these systems are safe and monitored and overseen throughout various stages in their life cycles.

We’ve barely scratched the surface on this. We’re talking about evaluation today. We’re not really talking so much about monitoring, observability, explainability, and oversight once a system is in the wild. But I do think that all of that infrastructure and structure needs to come.

Tim Scarfe

So you both work for Prolific. Now, I think it’s fair to say that what you folks are trying to do—as per that Apollo Research paper that you shared with me, Enzo, which is talking about how we need to have a science of evaluations—is... They were sketching out a maturity curve. So we’re kind of in the Wild West at the moment.

9. Human Values Need Humans

Enzo

What we do on our platform is try to get as many humans into this process as possible. I know there are a lot of efforts to take humans out of the loop, rightfully so, right? It all has its time and place. But when we talk about alignment, and specifically value alignment, then we need to, at least in some capacity, be able to capture the breadth of humanity.

LMArena is opt-in. People can go there at any stage, interact with models, and select their preference of which is better, according to no reason aside from selecting one over the other.

And there's no control for any population. So we can't really draw back, I guess, a causal relationship of what factors are at play here in a population. You could speculate that the kind of people who might participate in Chatbot Arena are biased to a very large degree, right? It's good. You can almost say that Chatbot Arena might be representative of how the tech world is perceiving the validity of these models or the preference of these models.

Tim Scarfe

On this leaderboard illusion, that was Marzyeh Ghassemi, Sara Hooker, and Shivalika Singh at Coherent, along with a few other people. We did a video on that recently. It's become the de facto standard for benchmarking large language models, and it has so many problems. It's now, after this investment, worth $600 million.

There's the insane selection bias, the bias in sampling, and the private pools where folks can get more matches; then they can take that training data and fine-tune on it. Also, the foundation models from Google, Meta, xAI, and so on, just get given more matches. It's just incredibly unfair. And as you were just saying before, even when folks put their prompts in, something like 50% of the prompts are basically carbon copies of the last month.

Tim Scarfe

Mm-hmm.

Tim Scarfe

I think it was 25% exactly the same, and then I think the other 25% was nearly the same, like 95% cosine distance on the embeddings or something like that. To me, that is an example of a superficially good rank, but it's flawed in so many ways.

Sara Sabb

The benchmarking of Grok 4 is also really interesting on this—

Tim Scarfe

Yes.

Sara Sabb

Because I'd be very happy to give them a lot of props for some of the stuff they're trying to do out in the open. Grok 4 wiped the floor on every benchmark, right? Including Humanity's Last Exam. Usability experiments are revealing, leaving aside some of the more troubling findings, that it's not a model that feels really natural to use. So I think even in the best of cases, these benchmarking-led approaches to evaluation seem to be failing us so far.

Enzo

I think if we try to describe the eval space a bit more holistically, LMArena tackles a very specific part of it, right? On the one end of the spectrum, we have very technical evals that are effectively closed-ended solutions. It's effectively a benchmark with known outcomes. We can see whether these are hit or not. Ultimately, it's a measure of accuracy, factuality, or correctness, if you will, and completeness to some degree.

At the other end of the spectrum, we have full subjectivity, entirely down to individual preference, right? Chatbot Arena is somewhere in between, because you're not actually ranking or evaluating for one or the other. It is technically preference, but you don't know quite whether the preference is because it said something wrong, whether the formatting was off, whether it didn't hit the cultural relativity or sensitivity, or whether it wasn't adaptive enough. It doesn't tell you anything of the sort.

Yeah. So we've developed a leaderboard which we've joyfully called Humane, which is trying to address some of the limitations that were found with common leaderboards. It's the same principle, ultimately. Someone is able to have multi-turn conversations with models that are effectively blind, or blindly selected. We're doing some a priori corrections, so we know most of their demographic and socioeconomic backgrounds in advance. We're doing selection beforehand for the type of people that go into it.

We're giving feedback right then and there as someone interacts with the model. We're giving warnings when it's a low-effort ask or when it's potentially unsafe, and so on, just to, I guess, pre-sanitize some of the inputs.

Sara Sabb

We study the benchmarking based on the demographic stratification of the humans doing the evaluations. So you can see things emerge in the data, like people of this age range think this model is better on helpfulness, but people of that age range disagree, and similarly with ethnicity, gender, and other strata.

Tim Scarfe

Yeah, I was looking at that. So I filtered on age and then background and culture. Essentially, I had background and culture grouped by different ages and so on. There were some patterns, and it turns out that older people felt that the models were more aligned to their culture. I was thinking, why is that? Is it because the data the model was trained on was just older, so it was more aligned to older people?

Culture is a very abstract term, and even in that value-alignment paper that you spoke about, Sara, and also—we'll talk about the PRISM paper—what we try to do is come up with a rubric, and we use these abstract terms. Sometimes this can be problematic because I used to keep a kind of “How am I feeling today?” diary, and I would have all of these tags. I would come up with a different word every day because the previous words didn't quite sufficiently explain how I was feeling that particular day.

So if you say to loads of people, “How culturally aligned is this conversation?” people have quite a differential understanding of that, don't they? So what does that actually mean?

Sara Sabb

To take a broad stab at this, something that feels really stark to me in what you've just said, Tim, is that we are basically constructing these towers that always end up at the ontologies we have in the world. I think we are trying to construct sanitary, deterministic, scientific systems, including when we do evaluation. In the end, we find that we're trafficking in messy concepts, and I think that keeps on happening. But I think the lesson is that we have to keep trying, not sort of put evaluation in a box and say we don't do it.

Tim Scarfe

This comes up quite a lot on the show, actually, that there's a perspectivist or constructivist-relative idea of a concept, which is that all of these different people with their different experiences have a perspective on something, and then in the infosphere, this thing emerges. It's quite nebulous, and we roughly draw a boundary around it, and we say that that's the thing. But actually, it's a million pointers from different people to this cloud, and you can't really reduce it to a single thing.

Sara Sabb

100%. I'm really vibing with the way you've described that, and I think the very first thing we need to do is try to get perspective into the mix through representativeness, at the very least, right? Knowing that there isn't this single, very well-defined concept of, let's say, generosity or morality. At the very least, we need to get lots of credible and durable perspectives on these things in from the humans of the world. And I say at the very least, but actually that's a very tall order. The kinds of evaluation we're doing right now are so far from even that.

Tim Scarfe

The capital of Paris. Lots of consensus on that, I would hope. Is abortion illegal? And by the way, I got this from the PRISM paper by Hannah Kirk, so maybe we can bring that in as well. But there are so many things in our culture that we just don't agree on. How should we deal with that?

Enzo

The capital of Paris is something that is factually agreeable. We can all agree on it, right? So that's on one end of the spectrum. And then there are certain things, maybe more akin to what Constitutional AI is doing: a set of policies we should all also agree on, and we can evaluate objectively whether the policies are met, right?

But we don't need to ask every individual on this earth to come up with the policies. This is something that a set of us, ideally a representative set of us, can agree on. And then further down the spectrum comes more preference data, or preferences in general, where it becomes more and more individualized, right? Even marginal groups, or even smaller and smaller representative groups, are becoming more relevant, all the way down to the individual when we talk about personalization, right?

Personalization is not something that we should build into the models; that can be handled with context, for example, right? But the capital of France being Paris is a fact, a fact that can be trained in. Something like a policy is also something that we can—or adherence to policy is something that we can also train in. But as we cut down further and further, all the way down to the individual, this is something we have to effectively stop at some point. And then it comes down to the person.

Tim Scarfe

There are all of these humans out there, these diverse humans, and they know a lot of things. How do we get useful signals from those folks? We need to do verification, right? It's not as easy as it sounds because sometimes, if I'm crowdsourcing a load of information, I probably shouldn't be weighting what those folks say too much because maybe, in some cases, I know that they're experts, and in some cases, I don't. You've been looking at things like voting schemes and consensus schemes to try and denoise that information.

How are you doing that?

Enzo

Let me paint a bit of a broader picture. There are sort of 2 camps, if you will. When I first started in this, it was all about machine learning; now it's AI. But there are 2 camps: on the one hand, data is a really important part of the equation, and then there are the algorithms, if you will.

Most people focus on the algorithm. Most people agree that the data part is perhaps the less sexy part. That's what we focus on. That's what we're all about. There is a lot of agreement coming out of lots of recent papers, including the Constitutional AI paper, that quality trumps quantity. Yet most of the models these days have been trained on an enormous corpus of data with tons and tons of noise in it, right? Even things like RLHF are just a comparison between 2 outputs, with almost no reasoning as to why that is.

So how do we bring more quality into it? Even in your traditional machine learning model—a supervised model for a narrow target or something—back then, it was irrelevant who was reviewing whether something was a cat or a dog, for example. Anyone could do that, and you could trust the quality of that data to some degree, right? But now we're in a world where foundation models have massive amounts of capabilities, and we really need to question who is producing the data that we're training on, but also the data that we're evaluating on.

We're making very, very far-reaching decisions to evaluate whether a model is safe, whether a model is doing well on something, or whether it converges well in the training step. This is all based ultimately on something that most people consider ground truth. But what if that ground truth is inherently noisy? What if that ground truth is inherently susceptible to tons of variance because it's not the right people who have reviewed it, there's some bias in the order something has been reviewed, or even in the interface something is being reviewed in? There are so many little factors that influence the quality of these labels. It's actually kind of fun to try to unpick all of these different factors that go into it.

Sara

There's also a really interesting piece of work being done by a group called the Collective Intelligence Project. This group is asking groups of people from around the world their views on a variety of AI ethics, AI safety, and responsible AI topics. If you track the societal groupings, it seems you find a nice carving point for norms. Humans live in society, and they tend to share their cultural beliefs with their tribes.

I think that's why being able to stratify the data you gather for evaluation from people is quite powerful, actually. Then you have these durable strata that travel through time. If we talk about the Apollo Research maturity curve, I thought that was really interesting and really brings to the forefront that this is pretty high-stakes stuff. The analogy was drawn to aircraft safety—

Tim Scarfe

Yes.

Sara

—and the norms that airline bodies put in place becoming legally influential when something goes wrong.

Tim Scarfe

I took some notes on that Apollo Research. It was saying: What precisely does the evaluation measure? How large is the coverage of the evaluation? How robust, in general, are the results of the evals? What is the replicability and reliability of the evals? Are there any statistical guarantees? How accurate are the predictions about future systems? I guess this is what you're saying. This is the kind of maturity curve that we need.

Sara

I think we think that, but I'm not sure the industry thinks that. I think that's a really interesting thing. Some in the industry think that. I think people working at the very frontier of state-of-the-art models do believe that evals need to be rigorous and robust, and humans have to be in the loop.

But I think we are also at a very, very early stage of “break it and apologize later,” where I think a big swathe of our industry doesn't yet think that this sort of human-mediated evaluation is going to be important and perhaps will get in the way of innovation. I think that tension is really important to resolve as well.

Tim Scarfe

Some of these statistical guarantees made a lot of sense 5 years ago, when we had quite specific models that operate in a vertical domain or something like that. We now have these epically general language models that do all things to all people. Enzo, what you're saying is spot-on about how we need to curate the data. Sara, you're talking about stratifying the data to reduce representational bias. But these models will be used for so many purposes. What does that mean in a general sense?

It also reminded me of Elon Musk. There was a wonderful tweet saying, “Just imagine how the Anthropic safety team felt when Elon pushed Hitler straight to prod.” He was bemoaning afterward that, on V7 of the foundation model, they've started curating the data better. They started pulling out all of the racist data. But the thing is that there's still this fundamental subjectivity problem, because you can stratify and you can curate, but people from different cultures will be asking it different questions. How do you overcome that ambiguity?

Enzo

It's a very prevalent problem, right? Humans are diverse by nature. We're all unique at the very end. There's also, by the way, an entire big blind spot to all of this, because at the moment we have only been talking about humans directly interacting with models. There's a whole other side to the story, which is where models are making decisions about humans, and humans are influenced by them indirectly.

This is not where a human can give feedback directly or distill some form of preference, right? We need to ultimately capture, or be able to measure, the outcomes and the impact that these models have on humans. There's a really good example: you can check whether an AI system can make the right medical diagnosis, which we can check for factual accuracy.

Tim Scarfe

Mm.

Enzo

For that same system to also transfer that diagnosis to someone who is affected by it is an entirely different language in that moment, right? We should evaluate that differently. We can say the diagnosis was correct, but did it also have the correct impact on the patient in that moment?

I believe there are some countries where patients themselves are not allowed to receive the transcripts from the testing facility, on the chance of the patient misinterpreting the results, right?

Tim Scarfe

Oh.

Enzo

Our evaluations and our measurement here need to be nuanced enough to understand every part of the system individually. But the part that we ultimately care about is the one we solve for the end user here, the patient in the end. Did it elicit a meaningful change in them or not? That should be the ultimate goal, right? But this is not something we can directly optimize for; it's something we can ideally measure and build accountability on.

Sara

An interesting thing that I pulled out of that Value Compass paper by Shen et al. is the misalignment between what AIs think they are and what we think AIs are as people. AIs think, or aspire to be—I’m going to use provocative language—autonomous thinkers.

Tim Scarfe

Yeah.

Sara

The research found that humans don't want that, right? I think the reason I bring that up is that the way you evaluate a helpful system is, as you're saying, Tim, that impossible problem of covering every test case in an infinite algorithm, which we will never do.

But the way you evaluate a person, or a thinker, or an autonomous being—we have loads of examples in the world, right? Jury trials, for example. Nobody expects that the moral behavior of a human is all predetermined when they're born, and that we know exactly what right or wrong looks like in every case. We have loads of social structure for evaluating the behavior and agency of a person.

I do think that we have to stop equivocating between—Or maybe nobody's equivocating but me. I think we should assume we're building toward AGI or superintelligence or thinking creatures and work backward, as opposed to trying to box in systems.

Tim Scarfe

Yes. Because when I was reading that Anthropic paper about agentic misalignment, one of my thoughts was that these are individual agents. As you were just saying, Sara, in the real world, we know that we are fallible, which is why we build error-correction systems. One person can't press the red button to drop a nuke. We have juries and various forms of collectives to overcome individual errors.

I'm guessing we could do the same thing with AIs. I'm not sure what that Anthropic experiment would look like if you had a supervisor. In the KGB, they had an expression: “Trust but verify.” You can almost have a supervisor agent, and you can have a committee of agents. But then we're getting into even murkier territory, right? Because we're building these inscrutable things.

And I am amenable, by the way, to this idea of loss of control, which is that we start to build systems on top of systems on top of systems, and it's a little bit like the power station. You can't just turn off a power station when we start to increasingly rely on all of this stuff. But, in principle, do you think that building some kind of agentic network could overcome some of these alignment problems?

Sara

I think certainly there will be networks and conditional layers when it comes to evaluation, oversight, and monitoring. We're already seeing it. This is very much mainstream already, right? Your evaluations of your model will be done by automated benchmarks, an LLM as a judge, some kind of oracle, or a reward function or something. And then there will be—whether it's considered human evaluation or not—some human who will verify something along the chain, and there's this kind of orchestration that's emerging between machines and people in the space of evaluations and oversight. So I think it feels pretty uncontroversial to say that there will be layered and orchestrated approaches like that. But what that looks like when you push the dial to 12, I'm not sure.

Tim Scarfe

Well, Enzo and Sara, this has been absolutely amazing. Thank you so much for joining us today.

Enzo

Thank you for having us.

Sara

Thank you, Tim. Thank you, Enzo.