No Priors 第135期｜与 Humans& 创始人 Eric Zelikman 对谈

Eric, thank you so much for doing this. You've had an amazing impact as a researcher, including starting from your time at Stanford. I want to hear about that, but first, tell us how you got interested in machine learning at all.

I guess, going back really far, I've been motivated by this question: You have all of these people out there who have all of these things that they're really talented in, and all of these things that people are really passionate about. There's just so much talent out there, and I've always been a little bit disappointed that so much of that talent doesn't get used just because everyone has circumstances and situations where they can't actually pursue those things. And so, for me, AI—

All of humanity is not living up to its full potential. I mean—

And then you've got to do AI.

I mean, the thing I've always been excited about is: How do you actually build this technology that frees people up to do the things that they're passionate about? How do you basically allow people to focus on those things? Originally, I thought of automation as the most natural way of doing that. You automate away the parts that people don't want to do, and that frees people up to do the things that they do want to do.

But I guess I realized increasingly that it's actually pretty complex. You have to understand—if you want to empower people to do what they want to do—you have to really understand what people actually want to do. Building systems that understand people's goals and outcomes is actually really hard.

Did you have this human-centric perspective when you were choosing research problems to work on originally?

I guess at the very beginning, when I was choosing research problems to work on, it was more about increasing capability at first.

So it was more about increasing capability at first?

Yeah. I think for me, when I looked at AI, or language models, back in 2021 or whatever, I was like, these things aren't very smart; they can't do that much. There was some early work that showed, for example, you could use chain-of-thought to get models to answer more intelligently, but it was still only a small-step improvement at that time. The benefit of that was as much as you could really get with just prompting. Back then, I was thinking, okay, how do you actually make them half-decent at solving these harder problems?

Can you give a broad intuition for STaR? We have everything from a researcher audience to a businessperson audience here.

I guess the intuition is, if you have a model and it's able to solve these basic—these slightly harder—questions by thinking about them, then what if you actually teach it: “Hey, this solution that you came up with that got you to the right answer, good job.” Or, if the model didn't get the right answer, you basically don't reward it.

I guess the original version of STaR actually had—there wasn't a baseline at the time. We compared it to REINFORCE, which is this popular algorithm in reinforcement learning, a very simple policy-gradient thing. At the time, it was a very simple algorithm: You iteratively generate solutions; if the solutions get you to the right answer, you learn from them; if they don't, you don't. Then you just keep doing this as the model solves harder and harder problems and learns from harder and harder problems.

At what point in the research, if at all, were you surprised by how well it worked, or did you have some intuition for this being something scalable?

There was 1 experiment I remember doing, though this was quite a while ago. We looked at— I think it was n-digit addition or multiplication. One thing that was really interesting was that, back then, this was a task that was considered hard for language models. Of course, it was considered one of the examples of why they were still so stupid.

Exactly.

One of the really interesting things for me was that, as you trained for more and more iterations, the number of digits that it was able to do kept increasing.

Okay.

I think this was one of those big surprises for me: There was no obvious plateau here.

And did you go directly from that to thinking, generally, this should scale?

I think I was generally interested in that. I think there were a few things, though. We observed that there was a bunch of the data that the model wasn't learning from, so we proposed another variant where we were like, “What if you actually take the ones where it fails and basically ask it to reason about why it should have gotten it right, and then train as if it got it right?”

This version was a way of extending beyond the parts of the data that it couldn't see. If you only train on positive examples, then you end up in this kind of local minimum where there's just no more data that it can actually solve. Back then, we were like, “What if we just show it the problems that it didn't solve and try to teach it from those?” But another thing that other work has done since then is: What if you just sample a lot? That also seems to work in those works.

STaR has become a broadly used part of the reasoning paradigm since you published it. Can you also describe what I think is sort of your last published work, Quiet-STaR?

Yeah. Quiet-STaR was kind of the last thing that I did back at Stanford, and it was really fun. We showed a few things that were cool. One of the main goals of that paper was to show that you could actually scale this up to pretraining scale by using pretraining-style data.

Now there are a bunch of works that have come out recently around RL pretraining and things like that. In some ways, that's similar to some of what we showed in the Quiet-STaR work. Instead of having question-answer pairs, if you just have arbitrary chunks of text, for example, and it tries to predict what's going to come next, which is the standard language-modeling objective, can you actually get models that more generally learn to reason?

One of the cooler things that I think is kind of overlooked about the original Quiet-STaR paper is that we showed a bunch of key improvements to the STaR paper that were necessary to actually do this kind of thing. One of those was showing that it's really valuable for this algorithm to be online.

Mhm.

Another was showing that it's really valuable to have a baseline where, for harder problems, you learn more, and for easier problems, you don't learn quite as much. I think there were a bunch of nuggets in there that, even at the time, I didn't fully think of as, “Oh, wow, that's actually a cool improvement over the original thing.”

You ended up going to xAI for several years, and you worked on a bunch of different paradigms: pretraining data for Grok 2, and then the overall reasoning recipe for Grok 3. I'm sure I'm missing things, but also tool use and agentic infrastructure for Grok 4. If you level-set us today, how smart are models? They can obviously do n-digit arithmetic at this point.

I guess in terms of IQ stuff, I'd say that if you're able to pose the problem very well—like some very advanced physics or math problem—I would say they're reasonably smart. I think a lot of the failures that people—

Give me a human comparison. What is “reasonably smart”?

I think it's hard to compare directly because it's very jagged.

Yeah.

It's true that some of these—for example, some of the Humanity's Last Exam questions that these models are able to solve—are genuinely things that are nontrivial for actual PhD researchers. I'm not saying they're open problems or anything, but they are pretty nontrivial.

Also, a lot of them are—one interesting category, and I spend a lot of time looking at the Humanity's Last Exam questions, is actually quite large. These are trick questions that require you to recognize an assumption. If you're familiar with the subject, you'll be like, “Oh, they're trying to get you to assume something.” But if you think more carefully about the problem, that assumption doesn't hold.

This turns out to be a bunch of those kinds of problems. I think they’re pretty smart, but they’re more tripped up by some of these tricky things. One of the core things is that they’re not smart emotionally, or on the level of actually understanding what people care about, or how to help people accomplish the things they care about.

I want to talk about this and your next mission. But just on this topic of jagged intelligence within the IQ domain, which I think almost everybody in the industry has been focused on until now, what would you recommend for people who aren’t researchers to develop some sort of intuition for that surface? That seems very important to making them useful.

Yeah. I guess one thing that’s really important to keep in mind is that the more context you can give the current generation of models, the better off you are. Their answers are super sensitive to whatever additional information you can give them.

I would generally say existing models are particularly good at handling questions that are easy to answer in closed form. If there’s a simple numerical answer to what you’re asking, or a simple way of choosing from a set of things, that makes it easier for the model. If you can imagine it being easy to check your answer, that actually makes it easier for the models.

Elad Gil

What do you think is the most dominant explanation for attempts to use models in very verifiable domains like code still failing at sophisticated tasks? Is it just that the wrong context has been fed to them? Is the context window simply not large enough to support the scratchpad and continual testing? Why, in those domains, what is the biggest challenge?

Part of it is that there’s a balance. When people want to give users these models, it’s actually important that they’re not annoyingly slow. There are a number of problems where, if you gave the models more time, they would actually be able to answer better.

For example, in the coding context, you have to be reasonably responsive. It depends on the kind of setup, right? If you look at products like OpenAI’s Codex, which is this longer-running background thing, versus Cursor, which is more interactive, you have a bit more luxury with those more-background approaches to tackle harder problems, I’d say.

Yeah, I think it’s a tricky question. A lot of things depend on how far the distribution of what you’re asking is from the distribution that the models were actually trained with. If you happen to be asking a problem that’s very similar to the kinds of problems it’s seen before, then it’ll do great. If you’re asking a problem that’s very out of domain, this question is hard to answer concretely unless you know basically what the RL data for these specific tasks is.

Elad Gil

Right, and today, obviously, none of the model, code-agent, or code-interface companies are going to release a capability map for you of what their RL data looks like, which would be very useful. Intuitively, if you look outside of the pretraining internet datasets, there are types of problems and types of codebases that are much further out of distribution. When engineers try in those scenarios, they get a dumb agent back, right?

Another thing that matters a lot is just how verifiable the things are that you’re trying to get the model to do. There’s been a ton of work out there on making models less dependent on verifiable rewards. There are lots of cool published papers, but I believe most people would say that there’s still a gap between how well these models perform on verifiable tasks versus nonverifiable tasks.

Yeah, absolutely. This is the last real question on IQ, but it’s where 90%+ of industry energy—literally energy and compute—is focused. How would you characterize where we are in scaling and the obvious opportunity to improve from here?

There are still meaningful dimensions of scaling that haven’t been fully explored in terms of IQ. I think there are a lot of cool efforts out there, and a lot of cool things that can still be done on the capabilities axis. I do think that once you start thinking about some of these new axes of scaling, it’s natural to realize that there are ways to do them that incorporate people, and there are ways to do them that leave people out more and more.

Mhm. And being very mindful of, “Oh, hey, I’m designing this new algorithm, and it’s going to scale the IQ of this model by X amount.”

Mhm.

If you effectively keep people in the loop, that’s actually a very active decision. So, in general, if you’re thinking about these things, that’s important. Wouldn’t it be fair to claim that the instinct of many labs is to try to get people out of the loop as much as possible from a scaling perspective because that’s very messy? If I want to recruit people to, for example, take complex reasoning traces from them on tasks that aren’t in distribution for me yet, that’s not as simple for an organization to execute on as more rollouts. Why is that important at all from a capabilities perspective?

I mean, that’s a good transition to what you’re doing.

Yeah, I’d say that the main thing is that as these models expand in terms of the horizon they’re automating, you have models—the recent or recentish IMO results are a good example of this—that go on for hours of reasoning without any human intervention. This has been an increasing measure of success for these labs.

For example, there’s the METR benchmark that everyone likes to share whenever there’s a new model. It’s, “Oh, we went from being able to have these models work for 2-hour tasks autonomously, without human intervention, to 2.5-hour tasks without human intervention.” There are obviously questions about what those numbers actually mean and how much we should take them at face value, but regardless, this has been the metric that people are looking at more and more to measure progress.

As we get these models that increasingly remove people from the interaction, you end up with people having less say in the things that get built. If you have a model that goes off and does its own thing for 8 hours and comes back to you with something that’s somewhat there, I think this is a weird regime where people probably feel less real agency over the things they’re building. I also anticipate that people will feel like they don’t really understand the things that are being built.

You know, I think that’s already true.

I think it’s already true.

Elad Gil

Twenty thousand lines of generated code looks good to me.

Yeah. It’s just like you make these PRs and they’re 100,000 lines of generated code, and I think in general this is going to be part of the trend.

So do you think it’s important to have humans in the loop in producing the output or the reasoning because the ceiling is higher when humans are in the loop, because it’s more efficient and we can error-correct when models are off path, or philosophically because people want that, or some combination of all 3?

Yeah, I think it’s probably some combination. Another thing that I think about is that the most natural thing to do as you automate away the existing set of tasks is to look at the world GDP, carve out the parts that are easiest to replace with these models, and target those things. Coding is an X-billion-dollar market, so let’s automate all of that. This other segment is an X-billion-dollar market, so let’s automate all of that.

But I actually think that if you empower people—if you have models that really understand what people are trying to accomplish and really support them in accomplishing those things—you have the potential to grow that pie instead of basically replacing all of those segments.

Mhm.

And in general, if the purpose of these models is to replace the person for this chunk of work, you end up with a lot less real innovation in what’s possible. If you actually have models that really understand what people’s goals are and really empower them more, you end up in a very different situation, because we’re going to push those capabilities into areas that are out of distribution for them.

Okay, cool. I think—is that accurate?

Yeah. No, I’d say so. I think that when I say I’d like to work on models that empower people instead of replacing them, people are like, “Oh, yeah, sure,” but I’d rather work on curing cancer or something. Obviously, that’s a really important goal, right? Building models that are able to solve humanity’s most difficult and most fundamental problems is incredibly important.

But I also think—and I’m sure that many of the researchers in the field disagree—that in the long run, we’ll see what plays out. I personally strongly believe that we’re much more likely to solve a lot of these fundamental human problems by working together, by building models that are really good at collaborating with large groups of people; that are really good at understanding different people’s goals, ambitions, and values; understanding different people’s weaknesses; and figuring out how to coordinate with these large groups of people to make everyone more effective.

And I think the vision of this AI that goes off on its own for 20 hours, does its own thing, and comes back with the answers to life, the universe, and everything—I think that this is less likely. I guess we’ll have to see, but I think it’s less likely.

So that goes to the fact that you’re starting a new company, Humans&, and I remember being actually quite fundamentally surprised, given all of your work on IQ, reasoning, coding, and scale, that you were interested in essentially EQ. You also thought of EQ—and tell me if this is a wrong characterization—as the emotional or interactive capabilities of models today have really shown up only in things like character or companionship tools, and you thought of it as also enablement from a productivity perspective, right? Tell me where this thread came from.

Yeah, I guess I’ve been thinking about this kind of stuff for some time now. Even back in my PhD, I think one of my less well-known works was actually about showing that you can train language models to simulate different kinds of students—

Right?

—and tests.

Yeah, yeah.

And by simulating students, you can actually design better tests for those students. That was a really cool finding: if you have models that are really good at modeling people, you can actually design systems that are better for people. This was something that I found really cool.

As we move toward the current capabilities frontier, it became more and more obvious that we have these incredibly smart models that are capable of so much, but they’re not used for anywhere near what they’re capable of. The role that they play in people’s lives is a lot less deep and a lot less positive than it could be. I spent a lot of time thinking, “Why is that? Why are these models not more deeply and positively integrated into people’s lives?”

It seemed like a really big part of it is that, fundamentally, these models don’t really understand people. They don’t understand people’s goals. They’re trained—I would say part of it is the general training paradigm that the field is in. It’s very, I would say, single-task-focused, or task-centric.

It’s ludicrous that all the benchmarks are still oriented this way.

Yeah, I mean—

—or most of them.

Even the ones that are—there are very few benchmarks out there that actually try to consider, “What if you actually have a person interacting with this model?” At best, you have some multi-turn benchmarks that try to simulate how an environment would respond differently to different inputs. But even that is still far from considering, “If you actually have this model interacting with a person for some amount of time, how does it actually affect that person’s life?”

It’s really remarkable that the field is so stuck in this task-centric regime, but I think it makes a lot of sense. One thing that I was told by some folks at Google is that one of the reasons is that it’s actually very useful for credit assignment. Being able to have benchmarks that are very easy to quantify and very easy to relate to some immediate thing means that you can say, “This team did 2% better than this team, so they deserve all of the resources,” or, “This team improved the benchmark by 10% while this team improved it by 5%, so let’s allocate accordingly.”

In general, that’s part of it. I think another part of it is more aligned with the easiest ways to train these models. It’s not easy to have these RL environments and stuff. You have lots of these companies popping up, obviously, that are trying to sell environments to different people, but—

Elad Gil

The most popular are, of course, in coding and computer use.

Yeah, rather than anything that requires simulating people.

Yeah, it’s not that surprising that we’re in this current regime. But what do models need to know about people? What capabilities are they either missing or have not been elicited from them?

The most fundamental thing is that models don’t understand the long-term implications of the things that they do and say. When you treat every turn of a conversation as its own game, and you basically think of it as, “Okay, you had this interaction; you’re done,” you need to make sure that this one response has all of the possible answers and all of the possible content. You don’t ever ask questions. You don’t ever try to clarify things. You don’t really tend to express uncertainty. You don’t tend to be proactive. You don’t tend to think about the long term.

You see a lot of even single-turn side effects of this kind of regime, and most of them are treated as their own problems to solve. You see issues that people highlight around sycophancy. You see issues—there was recent news around the psychosis stuff. There are a lot of these harmful effects that you get if you think about things in this very single-task or task-centric way.

But if you have models that consider the long-term implications—if I tell this person to start a company that sells gloves for catching ice cream, and I tell them that sounds like a good business idea, they might actually go and build that business and realize that it was not actually a good business idea. Having a model that can reason about the long-term implications of the things—

—and they won’t trust me anymore, and then they won’t pay for my compute.

Exactly.

And it’s all over.

Exactly.

No, I’m kidding. I think that’s really interesting. One of the very core principles we have at Conviction for how we make decisions is: What is the very long-term thing we want? If that is the customer, the founder in this case, an LP, or even us, it actually simplifies things quite a bit if you say we’re optimizing for a decade-plus versus this interaction.

Being single-turn versus multi-turn seems like a very different way to make decisions. It seems very hard to collect data about multi-turn human interactions, especially when you get to time. It’s actually analogous to a problem in biology: How do you study diseases that just take time to progress?

I think it’s a really fundamental question. There is actually some good academic work that has started to explore some of this. There’s some work recently around RL from human interaction. There’s a cool paper called CollabLLM that trains against simulation. There’s a lot of very cool work starting to explore this in academia, but in general, I would say there’s a lot less attention being paid to this kind of stuff in industry.

For most labs—and maybe this is a strong statement—I would say that for most labs, the human is the intermediate step until you have this fully automated system. Spending a lot of time optimizing things to be really good at understanding, interacting, and collaborating with people is almost like an intermediate thing you have to do until you get to this fully automated point.

Can you paint a picture of how, if we have models that better understand human objectives over different time scales and are good at interacting with humans, that is more integrated into your life 5 years from now?

Yeah, I think you don’t need to go that far out—

2 years.

But, yeah, I think you get a lot of behaviors that you currently don’t really see in these models. You have models that are much better at understanding how the things that you say in apps fit into the overall context of the things that you’re doing.

For example, if the model knows that you’re going to a wedding and then you ask it about booking hotels in Paris, it might consider, “Around the time of this event, I know that this user has all these things that are true about them.” A model that is generally able to think about how everything that you say fits into its understanding of that person would create a fundamentally different interaction.

Because right now, if you want to ask a question like that, you kind of have to dump all of this context in. You have to tell it, “Can you help me find a hotel in Paris? This is because I’m going to a wedding. I have these constraints. I have these people who need to be with me. It needs to do this. It needs to be...” You basically just have to dump all of the context that’s relevant to yourself into the model.

And.

Elad Gil

It’s also an expensive interaction.

Yeah.

Elad Gil

And something that most people won’t do.

Imagine if you had a friend where you had to reexplain everything about yourself to them every time you spoke.

Elad Gil

Yeah. Can you imagine if every time you interacted with someone, they remembered your name and maybe what you do, and just the really high-level sketch of your life? That friendship probably would not last very long.

Yeah, I think that’s kind of what the current models are.

Elad Gil

So you’d argue that any investment in memory that today’s models have is not that interesting or that core to their capabilities today?

I would say that memory is definitely a feature that has been underinvested in by the field. But I would say that it is kind of difficult to invest in memory in this very task-centric regime, because if you have a bunch of these independent tasks, the amount of information that each of those needs from other things that you’ve discussed is not all that high.

Because of the current paradigm, memory doesn’t end up being super useful in the training, and so these models are not particularly good at doing it.

Elad Gil

So, one other thing I said to you, I think out of a fear instinct more than anything else, but I feel like other people will have this reaction as well, is: I’m a unique snowflake. You can’t possibly simulate me and all of my self-consistency issues—between wanting to learn something today but not actually wanting to do the work, and wanting to eat cake but wanting to be in shape as well. We have different time scales and change our minds. I’m just constant distribution shift, and then you can’t possibly bring all of us under distribution. How do you react to that?

I think to a certain extent it’s probably a little bit true. It’s not easy to build these really good models of people. But I do think that the task for the model needs to be that it should be trying to do that. The model needs to actually be trying to learn all of this—trying to learn about you, trying to learn about the things that you care about. The actual objective of the model needs to be to kind of understand you, and it probably won’t be perfect. But boy, you can be a lot better than the current models.

Elad Gil

That seems totally reasonable, actually.

Yeah, I think it’s something that, as a field, we will probably get better at. I’m not going to pretend that I’m going to one-shot this problem, but I think even any serious effort gets you quite a long way.

Mhm.

Elad Gil

There is a cult sci-fi series about the Culture where you have these superintelligent Minds, and essentially all of the human and humanlike races live in a society where the Minds make most of the decisions. I forget the total humanoid population, but let’s say there are 30 or 40 Minds that are still relevant as people, in terms of perhaps being out of distribution or providing reasoning that the Minds cannot, and everybody else just lives in a world of abundance where they’re rock climbing and hanging out or whatever, and they do not produce. How is your view of abundance different?

Everyone kind of has things that they’re passionate about, and given the opportunity, I think people can do really cool things. I think the role of the model should be to allow people to do those really cool things that everyone kind of wants to do and accomplish those things that everyone kind of wants to accomplish. And I think we shouldn’t outsource all of the thinking and everything to these AI overlords or whatever. I think what we really want are models that are able to empower us.

Elad Gil

Amazing. Okay. Super-unique mission, amazing research work. You’re hiring an early team and getting a lot of compute. Who are you looking for on the recruiting side?

One thing that I think is actually probably a good thing that my previous company did is thinking of everyone, to some extent, as engineers. I’m looking for really strong infra folks who can build stuff. I’m looking for really strong researchers who can build stuff. I’m looking for really strong product folks who can build stuff. I’m looking for people who have thought a lot about users, who have thought a lot about memory on the research side.

On the infra side, I’m looking for people who have thought a lot about distributed systems and really fast inference, people who have been there to scale really big projects up. On the product side, I think people who are really creative about new modes of interaction, people who really deeply care about building beautiful, tasteful products.

Awesome. Thanks so much, Eric.

Thank you so much.

Congrats on the new company.

Thank you so much.